the Creative Commons Attribution 4.0 License.
the Creative Commons Attribution 4.0 License.
Statistical and machine learning methods for evaluating trends in air quality under changing meteorological conditions
- 1Institute for Data, Systems, and Society, Massachusetts Institute of Technology, Cambridge, USA
- 2Departments of Statistics and Data Science and Women’s Health, University of Texas, Austin, USA
- 3Department of Earth, Atmospheric, and Planetary Sciences, Massachusetts Institute of Technology, Cambridge, USA
- acurrent address: Department of Earth System Science, Stanford University, USA
- 1Institute for Data, Systems, and Society, Massachusetts Institute of Technology, Cambridge, USA
- 2Departments of Statistics and Data Science and Women’s Health, University of Texas, Austin, USA
- 3Department of Earth, Atmospheric, and Planetary Sciences, Massachusetts Institute of Technology, Cambridge, USA
- acurrent address: Department of Earth System Science, Stanford University, USA
Abstract. Evaluating the influence of anthropogenic emissions changes on air quality requires accounting for the influence of meteorological variability. Statistical methods such as multiple linear regression (MLR) models with basic meteorological variables are often used to remove meteorological variability and estimate trends in measured pollutant concentrations attributable to emissions changes. However, the ability of these widely-used statistical approaches to correct for meteorological variability remains unknown, limiting their usefulness in the real-world policy evaluations. Here, we quantify the performance of MLR and other quantitative methods using two scenarios simulated by a chemical transport model, GEOS-Chem, as a synthetic dataset. Focusing on the impacts of anthropogenic emissions changes in the US (2011 to 2017) and China (2013 to 2017) on PM2.5 and O3, we show that widely-used regression methods do not perform well in correcting for meteorological variability and identifying long-term trends in ambient pollution related to changes in emissions. The estimation errors, characterized as the differences between meteorology-corrected trends and emission-driven trends under constant meteorology scenarios, can be reduced by 30 %–42 % using a random forest model that incorporates both local and regional scale meteorological features. We further design a correction method based on GEOS-Chem simulations with constant emission input and quantify the degree to which emissions and meteorological influences are inseparable, due to their process-based interactions. We conclude by providing recommendations for evaluating the effectiveness of emissions reduction policies using statistical approaches.
Minghao Qiu et al.
Status: final response (author comments only)
-
RC1: 'Comment on acp-2022-232', Anonymous Referee #1, 26 Apr 2022
General Comments:
The idea of evaluating the performance of statistical and machine learning methods used to correct meteorological variability in emission trends using model results is an interesting one, since there have been many recent papers that have been written that use these methods on real-world data sets, without any real metric of their effectiveness in recovering the true trends in emissions.
Overall, this is a well-written paper with a set of carefully designed experiments to assess the performance of different statistical methods to determine meteorology corrected emission trends. The writing is high-quality, includes proper citations, and the figures are clear and easy to understand. I recommend publishing with a few minor corrections to improve the readability of the manuscript and comprehensibility for researchers without a background in statistics.
There were several places where more detail is warranted. Particularly in the description of the application of the different models, there was not a lot of detail and it was difficult to determine how these methods were applied to the data sets. This is important in assessing the conclusions of the paper, that an RF model is preferable to the other statistical methods, as the specific implementation of each method could have a significant impact on its performance. This is particularly true for the machine learning methods.
Specific Comments:
I found the discussion of causal methods in lines 65-76 to be slightly confusing in this paper, since the paper was not focusing on assessing causal links, but rather on testing counterfactuals—this link should have been more directly made clear, especially for the typical reader of this journal who doesn’t have a background in statistics/causal inference.
It would have been useful to have some type of overview cartoon for the different experiments and their relationship to the terms in equation 1— Table 1 was useful, but I had to read the paper through twice before the different simulations were clear to me, and it would have been helpful to have some visual aid for this.
Lines 146. What fraction of the meteorology-concentration relationship is due to changes in natural emissions?
Is there a clear separation between the training and test data for the Random Forest? As I understand this was in part the point of the double-machine learning method, but this should be more clearly spelled out. It would obviously be problematic if both the training and test data are used to evaluate the performance of the RF method to recover the emission-driven trends, as this would give an artificially good performance for this method.
Lines 170-176. Can you spell out what you mean by the “uncorrected” method here? Does that mean the term fi(Xit) is neglected in equation 1?
Section 3.4. How are the observations corrected? Are these using the meteorological correction models as determined from the GEOS-Chem model?
-
RC2: 'Comment on acp-2022-232', Benjamin Wells, 11 May 2022
Overall, I think this is a novel approach to the characterization of meteorological adjustments to air pollutant concentration trends and it is worth publishing. The use of air quality models to create a synthetic dataset by which meteorological adjustment methods can be evaluated has not been previously published that I am aware of. The selection of air quality model, statistical adjusment methods, and regions are appropriate and the presentation as a whole is clear and well thought-out. I have several minor concerns listed below that I think could be addressed through minor revisions.
- From the manuscript, it isn't clear how the linear trend estimates are calculated. Are they calculated using linear regression or some other method (e.g. Theil-Sen)?
- One of my biggest concerns is that the time period (7 years for U.S., 5 years for China) is too short to calculate meaningful trend estimates, and because of this, the trend estimates themselves may be a source of additional uncertainty. I fully understand the time and resource constraints of running chemical transport models, therefore, I'm not suggesting that this must be done, but merely that this is addressed as a limitation in the discussion. As a potential future research application, one could expand this type of analysis to a set of model runs evaluated over a longer time period, such as EPA's EQUATES series for 2002-2017 (https://www.epa.gov/cmaq/equates). The emissions from thes
- Another potential source of uncertainty in this application is the choice of meteorological year for the set of model runs where the meteorology is held constant. For example, as the authors describe, the 2011 year which was held constant for the U.S. was unusually hot and dry throughout the central region of the country. As a sensitivity analysis, I think it would be useful to see how much the predictions change by holding the meteorology constant using other years. For example, 2013 and 2014 were cooler and wetter than average for much of the U.S. For the current manuscript, I think it would be sufficient for the authors to address this point in the discussion.
- The choice of the June-August period for ozone does not capture the period of maximum ozone concentrations for all regions of the U.S. For example, the southeast U.S. typically sees its highest ozone concentrations in April or May, while California may experience peak ozone in September or October. I think a period of April to October would be sufficient to capture the peak ozone concentrations in all regions of the U.S. Again, nothing needs to be redone, but it would be useful to discuss this in the manuscript.
- As far as the interaction between meteorological and emissions-based effects, I agree that this is both a concern and a major challenge for any meteorological adjustment approach, and ultimately, it may not be possible to estimate the magnitude of these interactions. One major source of these interactions, especially in recent years, is wildfires: dry meteorological conditions contribute to more wildfires, and more wildfires contribute additional emissions. Wildfires are especially difficult to capture in a chemical transport model due to their unpredictability and the difficulty of characterizing their emissions. As wildfires can be an especially large contributor to PM2.5 concentrations, it would be useful to see them discussed in the context of their contribution to met/emissions interactions and overall uncertainty.
- While I don't plan to list them all here, I noticed several typos and minor grammatical errors in the manuscript while reading it.
-
AC1: 'Reply on RC2', Minghao Qiu, 16 May 2022
Dear editor and reviewer,
We appreciate the opportunity to respond to these detailed comments to further strengthen our paper. There is an incomplete sentence in one of the comments from reviewer 2 (Dr. Benjamin Wells). The current comment is "...such as EPA's EQUATES series for 2002-2017 (https://www.epa.gov/cmaq/equates). The emissions from thes". We are wondering if the reviewer has more to suggest and comment on this point and we would appreciate any clarification and the opportunity to fully address this question.
Thank you!
-
EC1: 'Reply on AC1', Anne Perring, 24 May 2022
Dr. Wells clarifies that the originally truncated point should read:
-One of my biggest concerns is that the time period (7 years for U.S., 5 years for China) is too short to calculate meaningful trend estimates, and because of this, the trend estimates themselves may be a source of additional uncertainty. I fully understand the time and resource constraints of running chemical transport models, therefore, I'm not suggesting that this must be done, but merely that this is addressed as a limitation in the discussion. As a potential future research application, one could expand this type of analysis to a set of model runs evaluated over a longer time period, such as EPA's EQUATES series for 2002-2017 (https://www.epa.gov/cmaq/equates). The emissions and meteorology inputs for the EQUATES model runs are available from this website.
-
EC1: 'Reply on AC1', Anne Perring, 24 May 2022
-
AC1: 'Reply on RC2', Minghao Qiu, 16 May 2022
Minghao Qiu et al.
Minghao Qiu et al.
Viewed
| HTML | XML | Total | BibTeX | EndNote | |
|---|---|---|---|---|---|
| 273 | 91 | 8 | 372 | 5 | 3 |
- HTML: 273
- PDF: 91
- XML: 8
- Total: 372
- BibTeX: 5
- EndNote: 3
Viewed (geographical distribution)
| Country | # | Views | % |
|---|
| Total: | 0 |
| HTML: | 0 |
| PDF: | 0 |
| XML: | 0 |
- 1