<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE article PUBLIC "-//NLM//DTD Journal Publishing DTD v3.0 20080202//EN" "https://jats.nlm.nih.gov/nlm-dtd/publishing/3.0/journalpublishing3.dtd">
<article xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink" article-type="research-article" dtd-version="3.0" xml:lang="en">
<front>
<journal-meta>
<journal-id journal-id-type="publisher">ACP</journal-id>
<journal-title-group>
<journal-title>Atmospheric Chemistry and Physics</journal-title>
<abbrev-journal-title abbrev-type="publisher">ACP</abbrev-journal-title>
<abbrev-journal-title abbrev-type="nlm-ta">Atmos. Chem. Phys.</abbrev-journal-title>
</journal-title-group>
<issn pub-type="epub">1680-7324</issn>
<publisher><publisher-name>Copernicus Publications</publisher-name>
<publisher-loc>Göttingen, Germany</publisher-loc>
</publisher>
</journal-meta>
<article-meta>
<article-id pub-id-type="doi">10.5194/acp-14-11791-2014</article-id>
<title-group>
<article-title>&lt;i&gt;De praeceptis ferendis&lt;/i&gt;: good practice in multi-model ensembles</article-title>
</title-group>
<contrib-group><contrib contrib-type="author" xlink:type="simple"><name name-style="western"><surname>Kioutsioukis</surname>
<given-names>I.</given-names>
<ext-link>https://orcid.org/0000-0002-4653-8442</ext-link>
</name>
<xref ref-type="aff" rid="aff1">
<sup>1</sup>
</xref>
<xref ref-type="aff" rid="aff2">
<sup>2</sup>
</xref>
</contrib>
<contrib contrib-type="author" xlink:type="simple"><name name-style="western"><surname>Galmarini</surname>
<given-names>S.</given-names>
<ext-link>https://orcid.org/0000-0002-0321-152X</ext-link>
</name>
<xref ref-type="aff" rid="aff1">
<sup>1</sup>
</xref>
</contrib>
</contrib-group><aff id="aff1">
<label>1</label>
<addr-line>European Commission, Joint Research Center, Institute for Environment and Sustainability, Ispra (VA), Italy</addr-line>
</aff>
<aff id="aff2">
<label>2</label>
<addr-line>Laboratory of Atmospheric Physics, Physics Department, University of Patras, Greece</addr-line>
</aff>
<pub-date pub-type="epub">
<day>11</day>
<month>11</month>
<year>2014</year>
</pub-date>
<volume>14</volume>
<issue>21</issue>
<fpage>11791</fpage>
<lpage>11815</lpage>
<permissions>
<copyright-statement>Copyright: &#x000a9; 2014 I. Kioutsioukis</copyright-statement>
<copyright-year>2014</copyright-year>
<license license-type="open-access">
<license-p>This work is licensed under the Creative Commons Attribution 3.0 Unported License. To view a copy of this licence, visit <ext-link ext-link-type="uri"  xlink:href="https://creativecommons.org/licenses/by/3.0/">https://creativecommons.org/licenses/by/3.0/</ext-link></license-p>
</license>
</permissions>
<self-uri xlink:href="https://acp.copernicus.org/articles/14/11791/2014/acp-14-11791-2014.html">This article is available from https://acp.copernicus.org/articles/14/11791/2014/acp-14-11791-2014.html</self-uri>
<self-uri xlink:href="https://acp.copernicus.org/articles/14/11791/2014/acp-14-11791-2014.pdf">The full text article is available as a PDF file from https://acp.copernicus.org/articles/14/11791/2014/acp-14-11791-2014.pdf</self-uri>
<abstract>
<p>Ensembles of air quality models have been formally and empirically shown to
outperform single models in many cases. Evidence suggests that ensemble error
is reduced when the members form a diverse and accurate ensemble. Diversity
and accuracy are hence two factors that should be taken care of while
designing ensembles in order for them to provide better predictions.
Theoretical aspects like the bias–variance–covariance decomposition and the
accuracy–diversity decomposition are linked together and support the
importance of creating ensemble that incorporates both these elements. Hence,
the common practice of unconditional averaging of models without prior
manipulation limits the advantages of ensemble averaging. We demonstrate the
importance of ensemble accuracy and diversity through an inter-comparison of
ensemble products for which a sound mathematical framework exists, and
provide specific recommendations for model selection and weighting
&lt;i&gt;for multi-model ensembles&lt;/i&gt;. The sophisticated ensemble averaging
techniques, following proper training, were shown to have higher skill across
all distribution bins compared to solely ensemble averaging forecasts.</p>
</abstract>
<counts><page-count count="25"/></counts>
</article-meta>
</front>
<body/>
<back>
<ref-list>
<title>References</title>
<ref id="ref1">
<label>1</label><mixed-citation publication-type="other" xlink:type="simple">AMS (American Meteorological Society): Enhancing weather information with probability forecasts, Bull. Am. Meteorol. Soc., 83, 450–452, 2002.</mixed-citation>
</ref>
<ref id="ref2">
<label>2</label><mixed-citation publication-type="other" xlink:type="simple">Bishop, C. M.: Neural Networks for Pattern Recognition, Oxford University Press, New York, NY, USA, 1995.</mixed-citation>
</ref>
<ref id="ref3">
<label>3</label><mixed-citation publication-type="other" xlink:type="simple">Bretherton, C. S., Widmann, M., Dymnikov, V. P., Wallace, J. M., and Bladè, I.: The effective number of spatial degrees of freedom of a time-varying field, J. Climate, 12, 1990–2009, 1999.</mixed-citation>
</ref>
<ref id="ref4">
<label>4</label><mixed-citation publication-type="other" xlink:type="simple">Brown, G., Wyatt, J., Harris, R., and Yao, X.: Diversity creation methods: a survey and categorisation, J. Inform. Fusion, 6, 5–20, 2005.</mixed-citation>
</ref>
<ref id="ref5">
<label>5</label><mixed-citation publication-type="other" xlink:type="simple">Delle Monache L., Wilczak, J., McKeen, S., Grell, G., Pagowski, M., Peckham, S., Stull, R., McHenry, J., and McQueen, J.: A Kalman-filter bias correction method applied to deterministic, ensemble averaged, and probabilistic forecast of surface ozone, Tellus B, 60, 238–249, 2008.</mixed-citation>
</ref>
<ref id="ref6">
<label>6</label><mixed-citation publication-type="other" xlink:type="simple">Dennis, R., Fox, T., Fuentes, M., Gilliland, A., Hanna, S., Hogrefe, C., Irwin, J., Rao, S. T., Scheffe, R.,Schere, K., Steyn, D., and Venkatram, A.: A framework for evaluating of regional-scale numerical photochemical modeling systems, Environ. Fluid Mech., 10, 471–489,&lt;a href=&quot;http://dx.doi.org/10.1007/s10652-009-9163-2&quot;&gt;https://doi.org/10.1007/s10652-009-9163-2&lt;/a&gt;, 2010.</mixed-citation>
</ref>
<ref id="ref7">
<label>7</label><mixed-citation publication-type="other" xlink:type="simple">Dosio, A. and Paruolo, P.: Bias correction of the ENSEMBLES high-resolution climate change projections for use by impact models: Evaluation on the present climate, J. Geophys. Res., 116, D16106, &lt;a href=&quot;http://dx.doi.org/10.1029/2011jd015934&quot;&gt;https://doi.org/10.1029/2011jd015934&lt;/a&gt;, 2011.</mixed-citation>
</ref>
<ref id="ref8">
<label>8</label><mixed-citation publication-type="other" xlink:type="simple">Errico R.: What Is an Adjoint Model?, Bull. Am. Meteorol. Soc., 78, 2577–2591, 1997.</mixed-citation>
</ref>
<ref id="ref9">
<label>9</label><mixed-citation publication-type="other" xlink:type="simple">Fern, X. Z. and Brodley, C. E.: Solving cluster ensemble problems by bipartite graph partitioning, in Proceedings of 21st International Conference on Machine Learning (ICML2004), Banff, Alberta, Canada, 4–8 July 2004, ACM Press, 281–288, 2004.</mixed-citation>
</ref>
<ref id="ref10">
<label>10</label><mixed-citation publication-type="other" xlink:type="simple">Galmarini, S., Bianconi, R., Klug, W., Mikkelsen, T., Addis, R., Andronopoulos, S., Astrup, P., Baklanov, A., Bartniki, J., Bartzis, J. C., Bellasio, R., Bompay, F., Buckley, R., Bouzom, M., Champion, H., D&apos;Amours, R., Davakis E., Eleveld, H., Geertsema, G. T., Glaab, H., Kollax, M., Ilvonen, M., Manning, A., Pechinger, U., Persson, C., Polreich, E., Potemski, S., Prodanova, M., Saltbones, J., Slaper, H., Sofiev, M. A., Syrakov D., Sørensen, J. H., Van der Auwera, L., Valkama, I., and Zelazny, R.: Ensemble dispersion forecasting–-Part I: concept, approach and indicators, Atmos. Environ., 38, 4607–4617, 2004.</mixed-citation>
</ref>
<ref id="ref11">
<label>11</label><mixed-citation publication-type="other" xlink:type="simple">Galmarini, S., Rao, S. T., and Steyn, D. G.: Preface, Atmos. Environ., 53, 1–3, 2012a.</mixed-citation>
</ref>
<ref id="ref12">
<label>12</label><mixed-citation publication-type="other" xlink:type="simple">Galmarini S., Bianconi, R., Appel, W., Solazzo, E., Mosca, S., Grossi, P., Moran, M., Schere, K., and Rao, S. T.: ENSEMBLE and AMET: Two systems and approaches to a harmonized, simplified and efficient facility for air quality models development and evaluation, Atmos. Environ., 53, 51–59, 2012b.</mixed-citation>
</ref>
<ref id="ref13">
<label>13</label><mixed-citation publication-type="other" xlink:type="simple">Galmarini, S., Kioutsioukis, I., and Solazzo, E.: E pluribus unum*: ensemble air quality predictions, Atmos. Chem. Phys., 13, 7153–7182, &lt;a href=&quot;http://dx.doi.org/10.5194/acp-13-7153-2013&quot;&gt;https://doi.org/10.5194/acp-13-7153-2013&lt;/a&gt;, 2013.</mixed-citation>
</ref>
<ref id="ref14">
<label>14</label><mixed-citation publication-type="other" xlink:type="simple">Geman S., E. Bienenstock, and R. Doursat.: Neural networks and the bias/variance dilemma, Neural Computat., 4, 1–58, 1992.</mixed-citation>
</ref>
<ref id="ref15">
<label>15</label><mixed-citation publication-type="other" xlink:type="simple">Gneiting T., Raftery, A. E., Westveld, A. H., and Goldman, T.: Calibrated Probabilistic Forecasting Using Ensemble Model Output Statistics and Minimum CRPS Estimation, Monthly Weather Rev., 133, 1098–1118, 2005.</mixed-citation>
</ref>
<ref id="ref16">
<label>16</label><mixed-citation publication-type="other" xlink:type="simple">Hamill T.: Interpretation of Rank Histograms for Verifying Ensemble Forecasts, Monthly Weather Revi., 129, 550–560, 2001.</mixed-citation>
</ref>
<ref id="ref17">
<label>17</label><mixed-citation publication-type="other" xlink:type="simple">Helton, J. C. and Davis, F. J.: Latin Hypercube Sampling and the Propagation of Uncertainty in Analyses of Complex Systems, Reliab. Engin. Sys. Safety, 81, 23–69, 2003.</mixed-citation>
</ref>
<ref id="ref18">
<label>18</label><mixed-citation publication-type="other" xlink:type="simple">Iman, R. L. and Conover, W. J.: A Distribution-Free Approach to Inducing Rank Correlation Among Input Variables, Communications in Statistics: Simulation and Computation, B11, 311–334, 1982.</mixed-citation>
</ref>
<ref id="ref19">
<label>19</label><mixed-citation publication-type="other" xlink:type="simple">Kalnay E.: Atmospheric modelling, data assimilation and predictability, Cambridge University Press, New York, 341 pp., 2003.</mixed-citation>
</ref>
<ref id="ref20">
<label>20</label><mixed-citation publication-type="other" xlink:type="simple">Kang D., Mathur, R., Rao, S. T., and Yu, S.: Bias adjustment techniques for improving ozone air quality forecasts, J. Geophys. Res., 113, D23308, &lt;a href=&quot;http://dx.doi.org/10.1029/2008JD010151&quot;&gt;https://doi.org/10.1029/2008JD010151&lt;/a&gt;, 2008.</mixed-citation>
</ref>
<ref id="ref21">
<label>21</label><mixed-citation publication-type="other" xlink:type="simple">Knutti, R., Abramowitz, G., Collins, M., Eyring, V., Gleckler, P. J., Hewitson, B., and Mearns, L.: Good practice guidance paper on assessing and combining multi model climate projections, in: Meeting Report of the Intergovernmental Panel on Climate Change Expert Meeting on Assessing and Combining Multi Model Climate Projections, Boulder, Colorado, USA 25–27, January 2010, edited by: Stocker, T. F., Qin, D., Plattner, G.-K., Tignor, M., and Midgley, P. M., IPCC Working Group I Technical Support Unit, University of Bern, Bern, Switzerland, 13 pp., 2010</mixed-citation>
</ref>
<ref id="ref22">
<label>22</label><mixed-citation publication-type="other" xlink:type="simple">Krishnamurti, T. N., Kishtawal, C. M., LaRow, T. E., Bachiochi, D. R., Zhang, Z., Williford, C. E., Gadgil, S., and Surendran, S.: Improved Weather and Seasonal Climate Forecasts from Multimodel Superensemble, Science, 285, 1548–1550, 1999.</mixed-citation>
</ref>
<ref id="ref23">
<label>23</label><mixed-citation publication-type="other" xlink:type="simple">Krogh, A. and Vedelsby, J.: Neural network ensembles, cross validation, and active learning, in: Advances in Neural Information Processing Systems, 7, 231–238, 1995.</mixed-citation>
</ref>
<ref id="ref24">
<label>24</label><mixed-citation publication-type="other" xlink:type="simple">Kuncheva, L. and Whitaker, C.: Measures of diversity in classifier ensembles, Machine Learn., 51, 181–207, 2003.</mixed-citation>
</ref>
<ref id="ref25">
<label>25</label><mixed-citation publication-type="other" xlink:type="simple">Leith C. E.: Theoretical skill of Monte Carlo forecasts, Monthly Weather Rev., 102, 409–418, 1974.</mixed-citation>
</ref>
<ref id="ref26">
<label>26</label><mixed-citation publication-type="other" xlink:type="simple">Lin M., K. Tang, and X. Yao: Selective negative correlation learning algorithm for incremental learning, in: Proceedings of the 2008 International Joint Conference on Neural Networks (IJCNN&apos;08), Hongkong, China, pp. 2526–2531, 1–6 June 2008.</mixed-citation>
</ref>
<ref id="ref27">
<label>27</label><mixed-citation publication-type="other" xlink:type="simple">Liu, Y. and Yao, X.: Ensemble learning via negative correlation, Neural Networks, 12, 1399–1404, 1999.</mixed-citation>
</ref>
<ref id="ref28">
<label>28</label><mixed-citation publication-type="other" xlink:type="simple">Malamud, B. D. and Turcotte, D. L.: Self-affine time series: measures of weak and strong persistence, J. Statist. Plann. Inf., 80, 173–196, 1999.</mixed-citation>
</ref>
<ref id="ref29">
<label>29</label><mixed-citation publication-type="other" xlink:type="simple">Mallet, V. and Sportisse, B.: Uncertainty in a chemistry-transport model due to physical parameterizations and numerical approximations: an ensemble approach applied to ozone modelling, J. Geophys. Res., 111, D01302, &lt;a href=&quot;http://dx.doi.org/10.1029/2005JD006149&quot;&gt;https://doi.org/10.1029/2005JD006149&lt;/a&gt;, 2006.</mixed-citation>
</ref>
<ref id="ref30">
<label>30</label><mixed-citation publication-type="other" xlink:type="simple">Markowitz, H.: Portfolio selection, J. Finance, 7, 77–91, 1952.</mixed-citation>
</ref>
<ref id="ref31">
<label>31</label><mixed-citation publication-type="other" xlink:type="simple">McKay M. D., Beckman, R. J., and Conover, W. J.: A Comparison of Three Methods for Selecting Values of Input Variables in the Analysis of Output from a Computer Code, Technometrics, 21, 239–245, 1979.</mixed-citation>
</ref>
<ref id="ref32">
<label>32</label><mixed-citation publication-type="other" xlink:type="simple">McKeen S., Wilczak, J., Grell, G., Djalalova, I., Peckham, S., Hsie, E.-Y., Gong, W., Bouchet, V., Menard, S., Moffet, R., McHenry, J., McQueen, J., Tang, Y., Carmichael, G. R., Pagowski, M., Chan, A., Dye, T., Frost, G., Lee, P., and Mathur, R.: Assessment of an ensemble of seven real-time ozone forecasts over eastern North America during the summer of 2004, J. Geophys. Res., 110, D21307, &lt;a href=&quot;http://dx.doi.org/10.1029/2005JD005858&quot;&gt;https://doi.org/10.1029/2005JD005858&lt;/a&gt;, 2005.</mixed-citation>
</ref>
<ref id="ref33">
<label>33</label><mixed-citation publication-type="other" xlink:type="simple">Molteni, F., Buizza, R., Palmer, T. N., and Petroliagis, T.: The new ECMWF ensemble prediction system: methodology and validation, Q. J. Roy. Meteor. Soc., 122, 73–119, 1996.</mixed-citation>
</ref>
<ref id="ref34">
<label>34</label><mixed-citation publication-type="other" xlink:type="simple">Potempski, S. and Galmarini, S.: Est modus in rebus: analytical properties of multi-model ensembles, Atmos. Chem. Phys., 9, 9471–9489, &lt;a href=&quot;http://dx.doi.org/10.5194/acp-9-9471-2009&quot;&gt;https://doi.org/10.5194/acp-9-9471-2009&lt;/a&gt;, 2009.</mixed-citation>
</ref>
<ref id="ref35">
<label>35</label><mixed-citation publication-type="other" xlink:type="simple">Pouliot, G., Pierce, T., Denier van der Gon, H., Schaap, M., Moran, M., and Nopmongcol, U.: Comparing Emissions Inventories and Model-Ready Emissions Datasets between Europe and North America for the AQMEII Project, Atmos. Environ., 53, 4–14, 2012.</mixed-citation>
</ref>
<ref id="ref36">
<label>36</label><mixed-citation publication-type="other" xlink:type="simple">Rao, S. T., Galmarini, S., and Puckett, K.: Air quality model evaluation international initiative (AQMEII): Advancing the state of the science in regional photochemical modelling and its applications, B. Am. Meteorol. Soc., 92, 23–30, 2011.</mixed-citation>
</ref>
<ref id="ref37">
<label>37</label><mixed-citation publication-type="other" xlink:type="simple">Riccio, A., Ciaramella, A., Giunta, G., Galmarini, S., Solazzo, E., and Potempski, S.: On the systematic reduction of data complexity in multi-model ensemble atmospheric dispersion modelling, J. Geophys. Res., 117, D05314, &lt;a href=&quot;http://dx.doi.org/10.1029/2011JD016503&quot;&gt;https://doi.org/10.1029/2011JD016503&lt;/a&gt;, 2012.</mixed-citation>
</ref>
<ref id="ref38">
<label>38</label><mixed-citation publication-type="other" xlink:type="simple">Schefzik, R., Thorarinsdottir, T. and Gneiting, T.: Uncertainty Quantification in Complex Simulation Models Using Ensemble Copula Coupling, Statist. Sci., 28, 616–640, 2013.</mixed-citation>
</ref>
<ref id="ref39">
<label>39</label><mixed-citation publication-type="other" xlink:type="simple">Schere, K., Flemming, J., Vautard, R., Chemel, C., Colette, A., Hogrefe, C., Bessagnet, B., Meleux, F., Mathur, R., Roselle, S., Hu, R.-M., Sokhi, R. S., Rao, S. T., and Galmarini, S.: Trace gas/aerosol boundary concentrations and their impacts on continental-scale AQMEII modeling domains, Atmos. Environ., 53, 38–50, 2012.</mixed-citation>
</ref>
<ref id="ref40">
<label>40</label><mixed-citation publication-type="other" xlink:type="simple">Solazzo, E., Bianconi, R., Vautard, R., Appel, K. W., Moran, M. D., Hogrefe, C., Bessagnet, B., Brandt, J., Christensen, J. H., Chemel, C., Coll, I., Denier van der Gon, H., Ferreira, J., Forkel, R., Francis, X. V., Grell, G., Grossi, P., Hansen, A. B., Jeričvićc, A., Kraljević, L., Miranda, A. I., Nopmongcol, U., Pirovano, G., Prank, M., Riccio, A., Sartelet, K. N., Schaap, M., Silver, J. D., Sokhi, R. S., Vira, J., Werhahn, J., Wolke, R., Yarwood, G., Zhang, J., Rao, S., and Galmarini, S.: Model evaluation and ensemble modelling of surface-level ozone in Europe and North America in the context of AQMEII, Atmos. Environ., 53, 60–74, 2012a.</mixed-citation>
</ref>
<ref id="ref41">
<label>41</label><mixed-citation publication-type="other" xlink:type="simple">Solazzo, E., Bianconi, R., Pirovano, G., Matthias, V., Vautard, R., Moran, M. D., Wyat Appel, K., Bessagnet, B., Brandt, J., Christensen, J. H., Chemel, C., Coll, I., Ferreira, J., Forkel, R., Francis, X. V., Grell, G., Grossi, P., Hansen, A. B., Miranda, A. I., Nopmongcol, U., Prank, M., Sartelet, K. N., Schaap, M., Silver, J. D., Sokhi, R. S., Vira, J., Werhahn, J., Wolke, R., Yarwood, G., Zhang, J., Rao, S. T., and Galmarini, S.: Operational model evaluation for particulate matter in Europe and North America in the context of AQMEII, Atmos. Environ., 53, 75–92, 2012b.</mixed-citation>
</ref>
<ref id="ref42">
<label>42</label><mixed-citation publication-type="other" xlink:type="simple">Solazzo, E., Riccio, A., Kioutsioukis, I., and Galmarini, S.: Pauci ex tanto numero: reduce redundancy in multi-model ensembles, Atmos. Chem. Phys., 13, 8315–8333, &lt;a href=&quot;http://dx.doi.org/10.5194/acp-13-8315-2013&quot;&gt;https://doi.org/10.5194/acp-13-8315-2013&lt;/a&gt;, 2013.</mixed-citation>
</ref>
<ref id="ref43">
<label>43</label><mixed-citation publication-type="other" xlink:type="simple">Stein, M.: Large Sample Properties of Simulations Using Latin Hypercube Sampling, Technometrics, 29, 143–151, 1987.</mixed-citation>
</ref>
<ref id="ref44">
<label>44</label><mixed-citation publication-type="other" xlink:type="simple">Tracton, M. S. and Kalnay, E.: Operational ensemble prediction at the National Meteorological Center: practical aspects, Weather Fore., 8, 379–398, 1993.</mixed-citation>
</ref>
<ref id="ref45">
<label>45</label><mixed-citation publication-type="other" xlink:type="simple">Ueda, N. and Nakano, R.: Generalization error of ensemble estimators, In Proceedings of International Conference on Neural Networks, Washington, DC, USA, 90–95, 3–6 June 1996.</mixed-citation>
</ref>
<ref id="ref46">
<label>46</label><mixed-citation publication-type="other" xlink:type="simple">Varotsos, C., Efstathiou, M., Tzanis, C., and Deligiorgi, D.: On the limits of the air pollution predictability; the case of the surface ozone at Athens, Greece, Environ. Sci. Pollut. Res., 19, 295–300, 2012.</mixed-citation>
</ref>
<ref id="ref47">
<label>47</label><mixed-citation publication-type="other" xlink:type="simple">Weigel, A., Knutt, R., Liniger, M., and Appenzeller, C.: Risks of model weighting in multimodel climate projections, J. Climate, 23, 4175–4191, 2010.</mixed-citation>
</ref>
<ref id="ref48">
<label>48</label><mixed-citation publication-type="other" xlink:type="simple">Zanda, M., Brown, G., Fumera, G., and Roli, F.: Ensemble Learning in Linearly Combined Classifiers Via Negative Correlation, Lecture Notes in Computer Science, 440–49, 2007.</mixed-citation>
</ref>
</ref-list>
</back>
</article>