Improving 3-day deterministic air pollution forecasts using machine learning algorithms

Johansson, Christer; Zhang, Zhiguo; Engardt, Magnuz; Stafoggia, Massimo; Ma, Xiaoliang

doi:https://doi.org/10.5194/acp-2023-38

Preprints

https://doi.org/10.5194/acp-2023-38

Preprints

08 Feb 2023

| 08 Feb 2023

Status: this preprint is currently under review for the journal ACP.

Improving 3-day deterministic air pollution forecasts using machine learning algorithms

Christer Johansson, Zhiguo Zhang, Magnuz Engardt, Massimo Stafoggia, and Xiaoliang Ma

Abstract. As air pollution is regarded as the single largest environmental health risk in Europe it is important that communication to the public is up-to-date, accurate and provides means to avoid exposure to high air pollution levels. Long- as well as short-term exposure to outdoor air pollution is associated with increased risks of mortality and morbidity. Up-to-date information on present and coming days’ air quality help people avoid exposure during episodes with high levels of air pollution. Air quality forecasts can be based on deterministic dispersion modelling, but to be accurate this requires detailed information on future emissions, meteorological conditions and process oriented dispersion modelling. In this paper we apply different machine learning (ML) algorithms – Random forest (RF), Extreme Gradient Boosting (XGB) and Long-Short Term Memory (LSTM) – to improve 1-, 2- and 3-day deterministic forecasts of PM₁₀, NO_x, and O₃ at different sites in Greater Stockholm, Sweden.

It is shown that the deterministic forecasts can be significantly improved using the MLs but that the degree of improvement of the deterministic forecasts depends more on pollutant and site than on what machine learning (ML) algorithm is applied. Deterministic forecasts of PM₁₀ is improved by the MLs through the input of lagged measurements and Julian day partly reflecting seasonal variations not properly parameterised in the deterministic forecasts. A systematic discrepancy by the deterministic forecasts in the diurnal cycle of NO_x is removed by the MLs considering lagged measurements and calendar data like hour of the day and weekday reflecting the influence of local traffic emissions. For O₃ at the urban background site the local photochemistry not properly accounted for by the relatively coarse Copernicus Atmosphere Monitoring Service ensemble model (CAMS) used here for forecasting O₃, but compensated using the MLs by taking lagged measurements into account. The machine learning models performed similarly well for the sites and pollutants. Performance measures like Pearson correlation, root mean square error (RMSE), mean absolute percentage error (MAPE) and mean absolute error (MAE), typically differed less than 30 % between ML models. At the urban background site, the deviations between modelled and measured concentrations (RMSE errors) are smaller than uncertainties in the measurements estimated according to recommendations by the Forum for Air Quality Modeling (FAIRMODE) in the context of the air quality directives. At the street canyon sites modelled errors are higher, and similar to measurement uncertainties. Further work is needed to reduce deviations between model results and measurements for short periods with relatively high concentrations (peaks). Such peaks can be due to a combination of non-typical emissions and unfavourable meteorological conditions and may be difficult to forecast. We have also shown that deterministic forecasts of NO_x at street canyon sites can be improved using MLs even if they are trained at other sites. For PM₁₀ this was only possible using LSTM.

An important aspect to consider when choosing ML is that the decision tree based models (RF and XGB) can provide useful output on the importance of features that is not possible using neural network models like LSTM, and also that training and optimisation is more complex with LSTM, which could be important to consider when selecting ML algorithm in an operational forecast system. A random forest model is now implemented operationally in the forecasts of air pollution and health risks in Stockholm. Development of the tuning process and identification of more efficient predictors may make forecast more accurate.

Received: 31 Jan 2023 – Discussion started: 08 Feb 2023

Christer Johansson et al.

Status: open (until 29 Mar 2023)

Post a comment Subscribe to comment alert

RC1: 'Comment on acp-2023-38', Anonymous Referee #1, 21 Mar 2023 reply

Review of “Improving 3-day deterministic air pollution forecasts using machine learning algorithms”
The authors present several ML models for air quality forecasting focusing on PM10, NOx and compare those results against deterministic forecasts. The dataset used to train ML models focuses on Stockholm. Overall, the authors show that the ML investigated seem to outperform the deterministic forecasts over the periods that the models were tested on, which were 1-, 2-, and 3-days in length periods. The ML models considered are very standard methods used widely in the literature, and are probably the appropriate starting points given the structure and size of training data. The authors additionally show some feature importance results.
In my opinion the length of this paper is too long and the results section needs to be shortened, the overall results summarized, and the key findings should be emphasized more. Both Table 2 and 4 could go to the Appendix as well as most of the figures in the results section. It might be easier to present/condense the text results in the tables as figures. I think the feature importance plots should be in the main text.
In the abstract it is wrongly stated that one cannot subject LSTMs to feature importance methods. A google search provides examples of how this can be performed. I think the authors need to investigate a gradient-based feature importance method for the LSTM and compare that model against the other models investigated, since the application of feature importance alongside the usage of ML models for longer-term horizon prediction is a main focus of their paper. Furthermore, the authors did not mention which importance method was used for the tree-based models. There are now a variety of methods available for these models, which often do not agree on the importance ranking. For the XGB model, how do other feature importance metrics (potentially such as the permutation and SHAP importance) compare to what was used?
Data
It is not clear in the paper how the data sources were combined and then split into training, and validation splits. Furthermore, it was not stated if any preprocessing of the data was performed (which probably needed to be carried out given the different ranges the input quantities cover). How large was the data set? The authors state that the ML models were trained on the same data; were the tree models trained on randomized (tabular) data, or was it split some other way (was the time-dependence preserved)? Figure 3 makes sense for the LSTM; however I think the authors could extend the figure to include some more schematic details pertaining to the tree-based models studied (and a schematic LSTM could be helpful as well).
Models
It becomes clear that there are in fact many models being used. How many relative to (presumably) Figure 4 (it currently says “Fel! Hittar inte referenskalla”)? The authors should probably add some comments about how this considerably complexifies the overall model pipeline, relative to one single model being used, and limits model(s) generalizability. Is this really a better solution relative to one single LSTM in terms of complexity?
There is no detail provided on model and training hyperparameters, and whether hyper parameter optimization was performed, for any of the models. How large was the LSTM (how many layers, layer size, what activation functions were used, etc.). Similarly, what were the XGB parameters used? Were all of the parameters guessed (or defaults used)? Overall, the models mostly look comparable. Given the results as they are in the current manuscript, it seems the obvious choice is RF, but I think the authors also need to compare linear regression to the other models, which should be considered the baseline ML model that the others need to beat. I don’t see much value in including the GAM (other than ruling it out).
Finally, it was not clear if cross-validation was performed and if the presented results show ensemble averages or something else. I think by now this is a very standard procedure, and it also provides an estimate of the uncertainty present in the models prediction capabilities given the training data set (for example, the tables should be presenting mean and variances for the ensemble). Given that there are so many models at play, it might be more useful to understand when models are more or less certain in their predictions.
Other
Most of the results in later figures show the mean of 1-,2-,3- day forecasts. How does the performance depend on day?
I’d rather the authors show the coefficient of determination (e.g. R2) alongside/rather than the Pearson coefficient.
Line 10: The deterministic predictions are used as models’ inputs but at which time? Is it the current prediction (and the models’ job is to correct CAMS?).
Throughout the paper the authors use “MLs” but this should probably read ML models for grammatical consistency.
In Figure 5 when there is a large drop just after September 20, what did the XAI method claim was the important feature(s). Is there anything different about the inputs to the model on that date?

Reply

Citation: https://doi.org/10.5194/acp-2023-38-RC1

Christer Johansson et al.

Data sets

ACP-2023-38 paper submission support: data for 3-days prediction of Air Quality using Machine Learning algorithms Johansson et al. https://zenodo.org/record/7576042#.Y9k9KHbMK72

Model code and software

ACP-2023-38 paper submission support: code for 3-days prediction of Air Quality using Machine Learning algorithms Johansson et al. https://zenodo.org/record/7576042#.Y9k9KHbMK72

Christer Johansson et al.

Viewed

Total article views: 399 (including HTML, PDF, and XML)

HTML	PDF	XML	Total	BibTeX	EndNote
309	81	9	399	3	1

HTML: 309
PDF: 81
XML: 9
Total: 399
BibTeX: 3
EndNote: 1

Views and downloads (calculated since 08 Feb 2023)

Month	HTML	PDF	XML	Total
Feb 2023	246	67	7	320
Mar 2023	63	14	2	79

Cumulative views and downloads (calculated since 08 Feb 2023)

Month	HTML	PDF	XML	Total
Feb 2023	246	67	7	320
Mar 2023	63	14	2	79

Viewed (geographical distribution)

Total article views: 406 (including HTML, PDF, and XML) Thereof 406 with geography defined and 0 with unknown origin.

Country	#	Views	%

Latest update: 27 Mar 2023

Short summary

Up-to-date information on present and coming days’ air quality help people avoid exposure to high levels of air pollution. We apply different machine learning models to significantly improve traditional forecasts of PM₁₀, NO_x, and O₃ in Stockholm, Sweden. It is shown that forecasts of all air pollutants are improved by through the input of lagged measurements and taking into account calendar information. The final modelled errors are substantially smaller than uncertainties in the measurements.


Total:	0
HTML:	0
PDF:	0
XML:	0