<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE article PUBLIC "-//NLM//DTD Journal Publishing DTD v3.0 20080202//EN" "https://jats.nlm.nih.gov/nlm-dtd/publishing/3.0/journalpublishing3.dtd">
<article xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink" article-type="research-article" dtd-version="3.0" xml:lang="en">
<front>
<journal-meta>
<journal-id journal-id-type="publisher">ACP</journal-id>
<journal-title-group>
<journal-title>Atmospheric Chemistry and Physics</journal-title>
<abbrev-journal-title abbrev-type="publisher">ACP</abbrev-journal-title>
<abbrev-journal-title abbrev-type="nlm-ta">Atmos. Chem. Phys.</abbrev-journal-title>
</journal-title-group>
<issn pub-type="epub">1680-7324</issn>
<publisher><publisher-name>Copernicus Publications</publisher-name>
<publisher-loc>Göttingen, Germany</publisher-loc>
</publisher>
</journal-meta>
<article-meta>
<article-id pub-id-type="doi">10.5194/acp-13-8315-2013</article-id>
<title-group>
<article-title>Pauci ex tanto numero: reduce redundancy in multi-model ensembles</article-title>
</title-group>
<contrib-group><contrib contrib-type="author" xlink:type="simple"><name name-style="western"><surname>Solazzo</surname>
<given-names>E.</given-names>
<ext-link>https://orcid.org/0000-0002-6333-1101</ext-link>
</name>
<xref ref-type="aff" rid="aff1">
<sup>1</sup>
</xref>
</contrib>
<contrib contrib-type="author" xlink:type="simple"><name name-style="western"><surname>Riccio</surname>
<given-names>A.</given-names>
<ext-link>https://orcid.org/0000-0001-7775-5565</ext-link>
</name>
<xref ref-type="aff" rid="aff2">
<sup>2</sup>
</xref>
</contrib>
<contrib contrib-type="author" xlink:type="simple"><name name-style="western"><surname>Kioutsioukis</surname>
<given-names>I.</given-names>
</name>
<xref ref-type="aff" rid="aff1">
<sup>1</sup>
</xref>
<xref ref-type="aff" rid="aff3">
<sup>3</sup>
</xref>
</contrib>
<contrib contrib-type="author" xlink:type="simple"><name name-style="western"><surname>Galmarini</surname>
<given-names>S.</given-names>
<ext-link>https://orcid.org/0000-0002-0321-152X</ext-link>
</name>
<xref ref-type="aff" rid="aff1">
<sup>1</sup>
</xref>
</contrib>
</contrib-group><aff id="aff1">
<label>1</label>
<addr-line>European Commission, Joint Research Centre, Institute for Environment and Sustainability, Ispra, Italy</addr-line>
</aff>
<aff id="aff2">
<label>2</label>
<addr-line>Department of Applied Science, University of Naples &quot;Parthenope&quot;, Napoli, Italy</addr-line>
</aff>
<aff id="aff3">
<label>3</label>
<addr-line>Region of Central Macedonia, Thessaloniki, Greece</addr-line>
</aff>
<pub-date pub-type="epub">
<day>22</day>
<month>08</month>
<year>2013</year>
</pub-date>
<volume>13</volume>
<issue>16</issue>
<fpage>8315</fpage>
<lpage>8333</lpage>
<permissions>
<copyright-statement>Copyright: &#x000a9; 2013 E. Solazzo et al.</copyright-statement>
<copyright-year>2013</copyright-year>
<license license-type="open-access">
<license-p>This work is licensed under the Creative Commons Attribution 3.0 Unported License. To view a copy of this licence, visit <ext-link ext-link-type="uri"  xlink:href="https://creativecommons.org/licenses/by/3.0/">https://creativecommons.org/licenses/by/3.0/</ext-link></license-p>
</license>
</permissions>
<self-uri xlink:href="https://acp.copernicus.org/articles/13/8315/2013/acp-13-8315-2013.html">This article is available from https://acp.copernicus.org/articles/13/8315/2013/acp-13-8315-2013.html</self-uri>
<self-uri xlink:href="https://acp.copernicus.org/articles/13/8315/2013/acp-13-8315-2013.pdf">The full text article is available as a PDF file from https://acp.copernicus.org/articles/13/8315/2013/acp-13-8315-2013.pdf</self-uri>
<abstract>
<p>We explicitly address the fundamental issue of member diversity in
multi-model ensembles. To date, no attempts in this direction have been documented
within the air quality (AQ) community despite the extensive use of
ensembles in this field. &lt;i&gt;Common biases&lt;/i&gt; and &lt;i&gt;redundancy&lt;/i&gt; are the two issues directly deriving from lack of
independence, undermining the significance of a multi-model ensemble, and
are the subject of this study. Shared, dependant biases among models do not
cancel out but will instead determine a biased ensemble. Redundancy derives from
having too large a portion of common variance among the members of the
ensemble, producing overconfidence in the predictions and underestimation of
the uncertainty. The two issues of common biases and redundancy are analysed
in detail using the AQMEII ensemble of AQ model results for four air
pollutants in two European regions. We show that models share large portions
of bias and variance, extending well beyond those induced by common inputs.
We make use of several techniques to further show that subsets of models can
explain the same amount of variance as the full ensemble with the advantage
of being poorly correlated. Selecting the members for generating skilful,
non-redundant ensembles from such subsets proved, however, non-trivial. We
propose and discuss various methods of member selection and rate the
ensemble performance they produce. In most cases, the full ensemble is
outscored by the reduced ones. We conclude that, although independence of
outputs may not always guarantee enhancement of scores (but this depends
upon the skill being investigated), we discourage selecting the members of
the ensemble simply on the basis of scores; that is, independence and skills
need to be considered disjointly.</p>
</abstract>
<counts><page-count count="19"/></counts>
</article-meta>
</front>
<body/>
<back>
<ref-list>
<title>References</title>
<ref id="ref1">
<label>1</label><mixed-citation publication-type="other" xlink:type="simple">Abramowitz, G.: Model Independence in multi-model ensemble prediction, Australian Meteorological and Oceanographic Journal, 59, 3–6, 2010.</mixed-citation>
</ref>
<ref id="ref2">
<label>2</label><mixed-citation publication-type="other" xlink:type="simple">Annan, J. D. and Hargreaves, J. C.: Reliability of the CMIP3 ensemble, Geophys. Res. Lett., 37, L02703, &lt;a href=&quot;http://dx.doi.org/10.1029/2009GL041994&quot;&gt;https://doi.org/10.1029/2009GL041994&lt;/a&gt;, 2010.</mixed-citation>
</ref>
<ref id="ref3">
<label>3</label><mixed-citation publication-type="other" xlink:type="simple">Borg, I. and Groenen, P.: Modern Multidimensional Scaling: theory and applications (2nd ed), Springer-Verlag, New York, 2005.</mixed-citation>
</ref>
<ref id="ref4">
<label>4</label><mixed-citation publication-type="other" xlink:type="simple">Bretherton, C. S., Widmann, M., Dymnikov, V. P., Wallace, J. M., and Bladè, I.: The effective number of spatial degrees of freedom of a time-varying field, J. Climate, 12, 1990–2009, 1999.</mixed-citation>
</ref>
<ref id="ref5">
<label>5</label><mixed-citation publication-type="other" xlink:type="simple">Brown, G., Wyatt, J. L., and Tino, P.: Managing diversity in regression ensembles, Journal of Machine Learning Research, 6, 1621–1650, 2005.</mixed-citation>
</ref>
<ref id="ref6">
<label>6</label><mixed-citation publication-type="other" xlink:type="simple">Cattell, R. B.: The scree test for the number of factors, Multivariate Behavioural Research, 1, 245–276, 1966.</mixed-citation>
</ref>
<ref id="ref7">
<label>7</label><mixed-citation publication-type="other" xlink:type="simple">Côté, J., Gravel, S., Méthot, A., Patoine, A., Roch, M., and Staniforth, A.: The Operational CMC–MRB Global Environmental Multiscale (GEM) Model. Part I: Design Considerations and Formulation, Mon. Weather Rev., 126, 1373–1395, 1998.</mixed-citation>
</ref>
<ref id="ref8">
<label>8</label><mixed-citation publication-type="other" xlink:type="simple">Cover, T. T.: The best two independent measures are not the two best, IEEE Trans. System Man. and Cybernetics, 4, 116–117, 1974.</mixed-citation>
</ref>
<ref id="ref9">
<label>9</label><mixed-citation publication-type="other" xlink:type="simple">Cover, T. and Thomas, J.: Elements of Information Theory, 2nd ed., Wiley-Interscience, Hoboken, NJ, 2006.</mixed-citation>
</ref>
<ref id="ref10">
<label>10</label><mixed-citation publication-type="other" xlink:type="simple">Dennis, R., Fox, T., Fuentes, M., Gilliland, A., Hanna, S., Hogrefe, C., Irwin, J., Rao, S. T., Scheffe, R., Schere, K., Steyn, D., and Venkatram, A.: A framework for evaluating regional-scale numerical photochemical modelling systems, Environ. Fluid Mech., 10, 471–489, &lt;a href=&quot;http://dx.doi.org/10.1007/s10652-009-9163-2&quot;&gt;https://doi.org/10.1007/s10652-009-9163-2&lt;/a&gt;, 2010.</mixed-citation>
</ref>
<ref id="ref11">
<label>11</label><mixed-citation publication-type="other" xlink:type="simple">Ding, C. and He, X.: K-means clustering via Principal component analysis, Proceedings of the 21st International Conference on Machine Learning, Banff, Canada, 2004.</mixed-citation>
</ref>
<ref id="ref12">
<label>12</label><mixed-citation publication-type="other" xlink:type="simple">Ding, C. and Peng, H.: Minimum Redundancy feature selection from microarray gene expression data, Journal of Bioinformatics and Computational Biology, 3, 185–205, 2005.</mixed-citation>
</ref>
<ref id="ref13">
<label>13</label><mixed-citation publication-type="other" xlink:type="simple">Elashoff, J. D., Elashoff, R. M., and Goldman, G. E.: On the choice of variables in classification problems with dichotomous variables, Biometrika, 54, 668–670, 1967.</mixed-citation>
</ref>
<ref id="ref14">
<label>14</label><mixed-citation publication-type="other" xlink:type="simple">Fern, X. Z. and Brodley, C. E.: Solving cluster ensemble problems by bipartite graph partitioning, in Proceedings of 21th International Conference on Machine Learning (ICML2004), 2004.</mixed-citation>
</ref>
<ref id="ref15">
<label>15</label><mixed-citation publication-type="other" xlink:type="simple">Fiore, A. M., Dentener, F. J., wild, O., Cuvelier, C., Schultz, M. G., Hess, P., Textor, C., Schulz, M., Doherty, R. M., Horowitz, L. W., MacKenzie, I. A., Sanderson, M. G., Shindell, D. T., Stevenson, D. S., Szopa, S., Van Dingenen, R., Zeng, G., Atherton, C., Bergmann, D., Bey, I., Carmichael, G., Collins, W. J., Duncan, B. N., Faluvegi, G., Folberth, G., Gauss, M., Gong, S., Hauglustaine, D., Holloway, T., Isaksen, I. S. A., Jacob, D. J., Jonson, J. E., Kaminski, J. W., Keating, T. J., Lupu, A., Marmer, E., Montanaro, V., Park, R. J., Pitari, G., Pringle, K. J., Pyle, J. A., Schroeder, S., Vivanco, M. G., Wind, P., Wojcik, G., Wu, S., and Zuber, A.: Multimodel estimates of intercontinental source-receptor relationships for ozone pollution, J. Geophys. Res., 114, D04301, &lt;a href=&quot;http://dx.doi.org/10.1029/2008JD010816&quot;&gt;https://doi.org/10.1029/2008JD010816&lt;/a&gt;, 2009.</mixed-citation>
</ref>
<ref id="ref16">
<label>16</label><mixed-citation publication-type="other" xlink:type="simple">Galmarini, S., Bianconi, R., Appel, W., Solazzo, E., Mosca, S., Grossi, P., Moran, M., Schere, K., and Rao, S. T.: ENSEMBLE and AMET: two systems and approaches to a harmonised, simplified and efficient assistance to air quality model developments and evaluation, Atmos. Environ., 53, 51–59, 2012.</mixed-citation>
</ref>
<ref id="ref17">
<label>17</label><mixed-citation publication-type="other" xlink:type="simple">Galmarini, S., Kioutsioukis, I., and Solazzo, E.: E pluribus unum*: ensemble air quality predictions, Atmos. Chem. Phys., 13, 7153–7182, &lt;a href=&quot;http://dx.doi.org/10.5194/acp-13-7153-2013&quot;&gt;https://doi.org/10.5194/acp-13-7153-2013&lt;/a&gt;, 2013.</mixed-citation>
</ref>
<ref id="ref18">
<label>18</label><mixed-citation publication-type="other" xlink:type="simple">Garaud, D. and Mallet, V.. Automatic calibration of an ensemble for uncertainty estimation and probabilistic forecast: Application to air quality, J. Geophys. Res., 116, D19304, &lt;a href=&quot;http://dx.doi.org/10.1029/2011JD015780&quot;&gt;https://doi.org/10.1029/2011JD015780&lt;/a&gt;, 2011.</mixed-citation>
</ref>
<ref id="ref19">
<label>19</label><mixed-citation publication-type="other" xlink:type="simple">Gleckler, P. J., Taylor, K. E., and Doutriaux, C.: Performance metrics for climate models, J. Geophys. Res., 113, D06104, &lt;a href=&quot;http://dx.doi.org/10.1029/2007JD008972&quot;&gt;https://doi.org/10.1029/2007JD008972&lt;/a&gt;, 2008.</mixed-citation>
</ref>
<ref id="ref20">
<label>20</label><mixed-citation publication-type="other" xlink:type="simple">Groenen, P. J. F and van de Velden, M.: Multidimensional Scaling. Erasmus University Rotterdam, Econometric Institute, Econometric Institute Report EI 2004–15, 2004.</mixed-citation>
</ref>
<ref id="ref21">
<label>21</label><mixed-citation publication-type="other" xlink:type="simple">Grömping, U.: Estimator of relative importance in linear regression based on variance decomposition, The American Statistician, 61, 139–147, 2007.</mixed-citation>
</ref>
<ref id="ref22">
<label>22</label><mixed-citation publication-type="other" xlink:type="simple">Guenther, A., Zimmerman, P., and Wildermuth, M.: Natural volatile organic compound emission rate estimates for US woodland landscapes, Atmos. Environ., 28, 1197–1210, 1994.</mixed-citation>
</ref>
<ref id="ref23">
<label>23</label><mixed-citation publication-type="other" xlink:type="simple">Guttman, L.: Some necessary conditions for common-factor analysis, Psychometrika, 19, 149–161, 1954.</mixed-citation>
</ref>
<ref id="ref24">
<label>24</label><mixed-citation publication-type="other" xlink:type="simple">Jolliffe, I.: Principal component analysis, Springer, 2nd edition, 2002.</mixed-citation>
</ref>
<ref id="ref25">
<label>25</label><mixed-citation publication-type="other" xlink:type="simple">Jong, J.-C. and Kotz, S.: On a relation between principal components and regression analysis, The American Statistician, 53, 349–351, 1999.</mixed-citation>
</ref>
<ref id="ref26">
<label>26</label><mixed-citation publication-type="other" xlink:type="simple">Jun, M., Knutti, R., and Nychka, D. W.: Local eigenvalue analysis of CMIP3 climate model errors, Tellus, 60, 992–1000, 2008.</mixed-citation>
</ref>
<ref id="ref27">
<label>27</label><mixed-citation publication-type="other" xlink:type="simple">Kaiser, H.: The application of electronic computers to factor analysis, Educational and Psychological Measurement, 20, 141, 1960.</mixed-citation>
</ref>
<ref id="ref28">
<label>28</label><mixed-citation publication-type="other" xlink:type="simple">Kaminski, J. W., Neary, L., Struzewska, J., McConnell, J. C., Lupu, A., Jarosz, J., Toyota, K., Gong, S. L., Côté, J., Liu, X., Chance, K., and Richter, A.: GEM-AQ, an on-line global multiscale chemical weather modelling system: model description and evaluation of gas phase chemistry processes, Atmos. Chem. Phys., 8, 3255–3281, &lt;a href=&quot;http://dx.doi.org/10.5194/acp-8-3255-2008&quot;&gt;https://doi.org/10.5194/acp-8-3255-2008&lt;/a&gt;, 2008.</mixed-citation>
</ref>
<ref id="ref29">
<label>29</label><mixed-citation publication-type="other" xlink:type="simple">Knutti, R.: The end of model democracy?, Climate Change, 102, 395–404, 2010.</mixed-citation>
</ref>
<ref id="ref30">
<label>30</label><mixed-citation publication-type="other" xlink:type="simple">Knutti, R., Furrer, R., Tebaldi, C., Cermak, J., and Meehl, G.: Challenges in Combining Projections from Multiple Climate Models, J. Climate, 23, 2739–2758, 2010.</mixed-citation>
</ref>
<ref id="ref31">
<label>31</label><mixed-citation publication-type="other" xlink:type="simple">Kong, W., Vanderburg, C. R., Gunshin, H., Rogers, J. T., and Huang, X.: A Review of independent component analysis application to microarray gene expression data, BioTechniques, 45, 501–520, 2008.</mixed-citation>
</ref>
<ref id="ref32">
<label>32</label><mixed-citation publication-type="other" xlink:type="simple">Legendre, P. and Legendre, L. F. J.: Numerical Ecology, Elsevier Science BV, Amsterdam, Chapter 11, 853 pp., 1998.</mixed-citation>
</ref>
<ref id="ref33">
<label>33</label><mixed-citation publication-type="other" xlink:type="simple">Liu, Y. and Yao, X.: Ensemble learning via negative correlation, Neural Networks, 12, 1399–1404, 1999.</mixed-citation>
</ref>
<ref id="ref34">
<label>34</label><mixed-citation publication-type="other" xlink:type="simple">McSweeney, C. F., Jones, R. G., and Booth, B. B. B.: Selecting ensemble members to provide regional climate change information, J. Climate, 25, 7100–7121, 2012.</mixed-citation>
</ref>
<ref id="ref35">
<label>35</label><mixed-citation publication-type="other" xlink:type="simple">Moesa, H. A., Dukka Bahadur, K. C., and Akutsu, T.: Efficient determination of cluster boundaries for analysis of gene expression profile data using hierarchical clustering and wavelet transformation, Genome Informatics, 16, 132–141, 2005.</mixed-citation>
</ref>
<ref id="ref36">
<label>36</label><mixed-citation publication-type="other" xlink:type="simple">Murtagh, F.: Complexities of Hierarchic Clustering Algorithms: the state of the art, Computational Statistics Quarterly, 1, 101–113, 1984.</mixed-citation>
</ref>
<ref id="ref37">
<label>37</label><mixed-citation publication-type="other" xlink:type="simple">Nenes, A., Pilinis, C., and Pandis, S.: ISORROPIA: a new thermodynamic equilibrium model for multicomponent inorganic aerosols, Aquat. Geochem., 4, 123–152, 1998.</mixed-citation>
</ref>
<ref id="ref38">
<label>38</label><mixed-citation publication-type="other" xlink:type="simple">Peng, H., Long, F., and Ding, C.: Feature selectionbased on mutual information: criteria of Max-dependency, Max-relevance, and Min-redundancy, IEEE Transactions on Pattern Analysis and Machine Intelligence, 27, 1226–1238, 2005.</mixed-citation>
</ref>
<ref id="ref39">
<label>39</label><mixed-citation publication-type="other" xlink:type="simple">Pennel, C. and Reichler, T.: On the effective numbers of climate models, J. Climate, 24, 2358–2367, 2011.</mixed-citation>
</ref>
<ref id="ref40">
<label>40</label><mixed-citation publication-type="other" xlink:type="simple">Pierce, D. W., Barnett, T. P., Santer, B. D., and Gleckler, P. J.: Selecting global climate models for regional climate change studies, P. Natl. Acad. Sci. USA, 106, 8441–8446, 2009.</mixed-citation>
</ref>
<ref id="ref41">
<label>41</label><mixed-citation publication-type="other" xlink:type="simple">Pirtle, Z., Meyer, R., and Hamilton, A.: What does it mean when climate models agree? A case for assessing independence among general circulation models, Environmental Science and Policy, 799, 351–361, 2010.</mixed-citation>
</ref>
<ref id="ref42">
<label>42</label><mixed-citation publication-type="other" xlink:type="simple">Potempski, S. and Galmarini, S.: Est modus in rebus: analytical properties of multi-model ensembles, Atmos. Chem. Phys., 9, 9471–9489, &lt;a href=&quot;http://dx.doi.org/10.5194/acp-9-9471-2009&quot;&gt;https://doi.org/10.5194/acp-9-9471-2009&lt;/a&gt;, 2009.</mixed-citation>
</ref>
<ref id="ref43">
<label>43</label><mixed-citation publication-type="other" xlink:type="simple">Potempski, S., Galmarini, S., Addis, R., Astrup, P., Bader, S., Bellasio, R., Bianconi, R., Bonnardot, F., Buckley, R., D&apos;Amours, R., van Dijk, A., Geertsema, G., Jones, A., Kaufmann, P., Pechinger, U., Persson, C., Polreich, C., Prodanova, M., Robertson, L., Sørensen, J., Syrakov, D.: Multi-model ensemble analysis of the ETEX-2 experiment, Atmos. Environ., 42, 7250–7265, 2008.</mixed-citation>
</ref>
<ref id="ref44">
<label>44</label><mixed-citation publication-type="other" xlink:type="simple">Rao, S. T., Galmarini, S., and Puckett, S.: Air quality model evaluation international initiative (AQMEII), B. Am. Meteorol. Soc., 92, 23–30, 2011.</mixed-citation>
</ref>
<ref id="ref45">
<label>45</label><mixed-citation publication-type="other" xlink:type="simple">Riccio, A., Ciaramella, A., Giunta, G., Galmarini, S., Solazzo, E., and Potempski, S.: On the systematic reduction of data complexity in multi-model ensemble atmospheric dispersion modelling, J. Geophys. Res., 117, D05314, &lt;a href=&quot;http://dx.doi.org/10.1029/2011JD016503&quot;&gt;https://doi.org/10.1029/2011JD016503&lt;/a&gt;, 2012.</mixed-citation>
</ref>
<ref id="ref46">
<label>46</label><mixed-citation publication-type="other" xlink:type="simple">Simpson, D., Guenther, A., Hewitt, C. N., and Steinbrecher, R.: Biogenic emissions in Europe. 1. Estimates and uncertainties, J. Geophys. Res., 100D, 22875–22890, 1995.</mixed-citation>
</ref>
<ref id="ref47">
<label>47</label><mixed-citation publication-type="other" xlink:type="simple">Solazzo, E., Bianconi, R., Vautard, R., Appel, K. W., Moran, M. D., Hogrefe, C., Bessagnet, B., 5 Brandt, J., Christensen, J. H., Chemel, C., Coll, I., van der Gon, H. D., Ferreira, J., Forkel, R., Francis, X. V., Grell, G., Grossi, P., Hansen, A. B., Jericevic, A., Kraljevic, L., Miranda, A. I., Nopmongcol, U., Pirovano, G., Prank, M., Riccio, A., Sartelet, K. N., Schaap, M., Silver, J. D., Sokhi, R. S., Vira, J., Werhahn, J., Wolke, R., Yarwood, G., Zhang, J., Rao, S. T., and Galmarini, S: Ensemble modelling of surface level ozone in Europe and North America in the context of AQMEI,  Atmos. Environ., 53, 60–74, 2012a.</mixed-citation>
</ref>
<ref id="ref48">
<label>48</label><mixed-citation publication-type="other" xlink:type="simple">Solazzo, E., Bianconi, R., Pirovano, G., Matthias, V., Vautard, R., Moran, M. D., Appel, K. W., Bessagnet, B., Brandt, J., Christensen, J. H., Chemel, C., Coll, I., Ferreira, J., Forkel, R., Francis, X. V., Grell, G., Grossi, P., Hansen, A. B., Hogrefe, C., Miranda, A. I., Nopmongco, U., Prank, M., Sartelet, K. N., Schaap, M., Silver, J. D., Sokhi, R. S., Vira, J., Werhahn, J., Wolke, R., Yarwood, G., Zhang, J., Rao, S. T., and Galmarini, S:  Operational Model evaluation for particulate matter In europe and North America in the context of AQMEII, Atmos. Environ., 53, 75–92, 2012b.</mixed-citation>
</ref>
<ref id="ref49">
<label>49</label><mixed-citation publication-type="other" xlink:type="simple">Solazzo, E., Bianconi, R., Pirovano, G., Moran, M. D., Vautard, R., Hogrefe, C., Appel, K. W., Matthias, V., Grossi, P., Bessagnet, B., Brandt, J., Chemel, C., Christensen, J. H., Forkel, R., Francis, X. V., Hansen, A. B., McKeen, S., Nopmongcol, U., Prank, M., Sartelet, K. N., Segers, A., Silver, J. D., Yarwood, G., Werhahn, J., Zhang, J., Rao, S. T., and Galmarini, S.: Evaluating the capability of regional-scale air quality models to capture the vertical distribution of pollutants, Geosci. Model Dev., 6, 791–818, &lt;a href=&quot;http://dx.doi.org/10.5194/gmd-6-791-2013&quot;&gt;https://doi.org/10.5194/gmd-6-791-2013&lt;/a&gt;, 2013.</mixed-citation>
</ref>
<ref id="ref50">
<label>50</label><mixed-citation publication-type="other" xlink:type="simple">Stewart, D. K. and Love, W. A.: A General Canonical Correlation Index, Psychol. Bullet., 70, 160–163, 1968.</mixed-citation>
</ref>
<ref id="ref51">
<label>51</label><mixed-citation publication-type="other" xlink:type="simple">Talagrand, O., Vautard, R., and Strauss, B.: Evaluation of probabilistic prediction systems, paper presented at aa seminar on predictability, Eur. cent. For Medium Weather Forecasting, Reading (UK), 1998.</mixed-citation>
</ref>
<ref id="ref52">
<label>52</label><mixed-citation publication-type="other" xlink:type="simple">Tebaldi, C. and Knutti, R.: The use of multi-model ensemble in probabilistic climate projections, Philos. Tr. Roy. Soc., 365A, 2053–2075, 2007.</mixed-citation>
</ref>
<ref id="ref53">
<label>53</label><mixed-citation publication-type="other" xlink:type="simple">Tinsley, H. E. A. and Brown, S. D.: Handbook of applied multivariate statistics and mathematical modeling, Academic Press, California (USA), 334, 338, 2000.</mixed-citation>
</ref>
<ref id="ref54">
<label>54</label><mixed-citation publication-type="other" xlink:type="simple">Toussaint, G. T.: Note on optimal selection of independent binary valued features for pattern recognition. IEEE Transactions on Information Theory, Vol. IT-17, 618, 1971.</mixed-citation>
</ref>
<ref id="ref55">
<label>55</label><mixed-citation publication-type="other" xlink:type="simple">Van Loon, M., Vautard, R., Schaap, M., Bergstrom, R., Bessagnet, B., Brandt, J., Builtjes, P. J., H., Christensen, J. H., Cuvelier, C., Graff, A., Jonson, J. E., Krol, M., Langner, J., Roberts, P., Rouil, L., Stern, R., Tarrason, L., Thunis, P., Vignati, E., White, L., and Wind, P.:  Evaluation of long-term ozone simulations from seven regional air quality models and their ensemble average, Atmos. Environ., 41, 2083–2097, 2007.</mixed-citation>
</ref>
<ref id="ref56">
<label>56</label><mixed-citation publication-type="other" xlink:type="simple">Vautard, R., Schaap, M., Bergström, R., Bessagnet, B., Brandt, J., Builtjes, P. J. H., Christensen, J. H., Cuvelier, C., Foltescu, V., Graf, A., Kerschbaumer, A., Krol, M., Roberts, P., Rouïl, L., Stern, R., Tarrason, L., Thunis, P., Vignati, E., and Wind, P.: Skill and uncertainty of a regional air quality model ensemble, Atmos. Environ., 43, 4822–4832, 2009.</mixed-citation>
</ref>
<ref id="ref57">
<label>57</label><mixed-citation publication-type="other" xlink:type="simple">Vautard, R., Moran, M. D., Solazzo, E., Gilliam, R. C., Matthias, V., Bianconi, R., Chemel, C., Ferreira, J., Geyer, B., Hansen, A. B., Jericevic, A., Prank, M., Segers, A., Silver, J. D., Werhahn, J., Wolke, R., Rao, S. T., and Galmarini, S.: Evaluation of the meteorological forcing used for AQMEII air quality simulations, Atmos. Environ., 53, 15–37, 2012.</mixed-citation>
</ref>
<ref id="ref58">
<label>58</label><mixed-citation publication-type="other" xlink:type="simple">Yoon, S. and Kim, S.: Mutual information-based SVM-RFE for diagnostic classification of digitized mammograms, Pattern Recognition Letters, 30, 1489–1495, 2009.</mixed-citation>
</ref>
<ref id="ref59">
<label>59</label><mixed-citation publication-type="other" xlink:type="simple">Youness, G. and Saporta, G.: Comparing partitions of two sets of units based on the same variables, Adv Data Anal Classif., https://doi.org/I0.1007/s11634-009-0057-4, 2010.</mixed-citation>
</ref>
<ref id="ref60">
<label>60</label><mixed-citation publication-type="other" xlink:type="simple">Zhang, L., Gong, S., Padro, J., and Barrie, L.: A size-segregated particle dry deposition scheme for an atmospheric aerosol module, Atmos. Environ., 549–560, 2001.</mixed-citation>
</ref>
<ref id="ref61">
<label>61</label><mixed-citation publication-type="other" xlink:type="simple">Zuber, V. and Strimmer, K.: High-Dimensional Regression and variable selection using CAR scores, Statistical Applications in Genetics and Molecular Biology, 10, 1–25, 2011.</mixed-citation>
</ref>
</ref-list>
</back>
</article>