Recommendations on benchmarks for numerical air quality model applications in China – Part 2: Ozone and uncertainty analysis

Ground-level ozone (O₃) has emerged as a significant air pollutant in China, attracting increasing attention from both the scientific community and policymakers. Chemical transport models (CTMs) serve as crucial tools in addressing O₃ pollution, with frequent applications in predicting O₃ concentrations, identifying source contributions, and formulating effective control strategies. The accuracy and reliability of the simulated O₃ concentrations are typically assessed through model performance evaluation (MPE). However, the wide array of CTMs available, variations in input data, model setups, and other factors result in a broad range of differences between simulated and observed O₃ concentrations, highlighting the necessity of standardized benchmarks in O₃ evaluation.

Building upon our previous work, this study conducted a thorough literature review of CTM applications simulating O₃ in China from 2006 to 2021. A total of 216 relevant articles out of a total of 667 reviewed were identified to extract quantitative MPE results and key model configurations. From our analysis, two sets of benchmark values for six commonly used MPE metrics are proposed for CTM applications in China, categorized into “goal” benchmarks representing optimal model performance and “criteria” benchmarks representing achievable model performance across a majority of studies. It is recommended that the normalized mean bias (NMB) for hourly O₃ and daily 8 h maximum O₃ concentrations should ideally fall within ±15 % and ±10 %, respectively, to meet the goal benchmark. If the criteria benchmarks are to be met, the NMB should be within ±30 % and ±20 %, respectively. Moreover, uncertainties in O₃ predictions due to uncertainties in various model inputs were quantified using the decoupled direct method (DDM) in a commonly used CTM. For the simulation period of June 2021, the total uncertainty of simulated O₃ ranged from 4 to 25 µg m⁻³, with anthropogenic volatile organic compound (AVOC) emissions contributing most to the uncertainty regarding O₃ in coastal regions and with O₃ boundary conditions playing a dominant role in the northwestern region. The proposed benchmarks for assessing simulated O₃ concentrations, in conjunction with our previous studies on PM_2.5 and other criteria air pollutants, represent a comprehensive and systematic effort to establish a model performance framework for CTM applications in China. These benchmarks aim to support the growing modeling community in China by offering a robust set of evaluation metrics and establishing a consistent evaluation methodology relative to the body of prior research, thereby helping to establish the credibility and reliability of CTM applications. These statistical benchmarks need to be periodically updated as models advance and as better inputs become available in the future.

1 Introduction

Tropospheric ozone (O₃) is a secondary air pollutant generated by complicated photochemical reactions involving nitrogen oxides (NO_x) and volatile organic compounds (VOCs) (Seinfeld and Pandis, 2016). Ozone has negative impacts on human health (Global Burden of Disease Collaborative Network, 2021), as well as on vegetation and ecosystem production (Ainsworth et al., 2012). Due to rapid economic development and fast industrialization and urbanization over the past several decades, China has experienced heavy haze pollution in winter and severe O₃ pollution in summer, with the latter extending into the late-winter haze season (Li et al., 2021). Despite efforts to reduce fine particulate matter (PM_2.5) and heavy-haze days (Wang et al., 2022; Bai et al., 2019; Chu et al., 2020), ground-level O₃ concentrations have continued to increase in recent years (Dang and Liao, 2019; Li et al., 2019; Liu et al., 2019a; Lu et al., 2020; Wang et al., 2017; Yao et al., 2023; Chen et al., 2023; Xu et al., 2023). The challenge in controlling O₃ pollution lies in the significant influences of meteorological conditions on O₃ formation and its nonlinear chemical relationship with precursors (Wang et al., 2022). In addition, O₃ pollution exhibits strong regional characteristics, necessitating regional-scale control efforts (Yang et al., 2021a).

Application of chemical transport models (CTMs) has become increasingly popular in addressing O₃-related issues in China (Yang and Zhao, 2023), providing insights into the role of local emissions and regional transport (Shen et al., 2022), sectoral contributions (Liu et al., 2020a), policy effectiveness (Liu et al., 2023b), and predictions of future O₃ levels (Yang and Zhao, 2023). Ensuring the representativeness of CTM simulations is crucial and can benefit from establishing performance standards or benchmarks to help put CTM results in context relative to the existing body of work. While other regions (e.g., the US and Europe) have proposed evaluation criteria for simulated O₃ (Emery et al., 2017), these may not be suitable for China. Several key factors necessitate the establishment of a tailored benchmark for model applications specific to China. Firstly, ozone concentrations in China have been significantly higher than those in the US and have shown a consistent upward trend (Chinese Society For Environmental Sciences, 2020). For instance, the fourth highest maximum daily 8 h average (4th MDA8) ozone concentration across 74 major cities in China increased from 189 µg m⁻³ (∼ 95 ppb) in 2013 to 236 µg m⁻³ (∼ 118 ppb) in 2019 compared to levels at or below 150 µg m⁻³ (∼ 75 ppb) in the US during the same period (Table S1 in the Supplement). Secondly, background ozone contributions exhibit different trends between China and other regions, with China experiencing a year-on-year increase, especially in urban areas (Chinese Society For Environmental Sciences, 2020). Thirdly, the mechanisms of ozone formation may differ between China and the US. However, a direct comparison of these formation regimes proves to be challenging as both countries encompass vast regions with distinct ozone dynamics. Jung et al. (2022) identified notable shifts in the western US, specifically from an NO_x-saturated regime to a transition regime (or from a transition regime to an NO_x-limited regime), while rural areas, especially in the eastern and southeastern US, have become increasingly sensitive to VOC emissions. In China, VOC-limited regimes were predominantly observed in the Beijing–Tianjin–Hebei (BTH), Yangtze River Delta (YRD), and Guangdong (GD) regions in 2013 (Zhang et al., 2024), whereas, in 2019, a significant transition from a VOC-limited regime to a transition regime was noted in the BTH area, accompanied by a reduction in VOC-limited areas within the YRD and GD regions. These disparities in ozone concentrations, background contributions, and formation mechanisms underscore the necessity of a customized benchmark for model applications in China, which is essential for appropriately addressing the unique challenges posed by ozone pollution within the country. Therefore, the increasing prevalence of CTM applications in China necessitates specific CTM benchmarks tailored to this region.

This study aims to develop customized CTM benchmarks for O₃ simulations in China, building upon our prior work that proposed evaluation indicators and benchmarks for simulating other criteria air pollutants (Huang et al., 2021; Zhai et al., 2024). A thorough literature review was conducted on O₃ simulations using CTMs from 2006 to 2021. Detailed information regarding O₃ performance was extracted and analyzed to recommended model performance evaluation (MPE) metrics and to propose benchmarks tailored to China. Furthermore, uncertainties in O₃ predictions due to various model inputs were quantified using the decoupled direct method (DDM) of sensitivity analysis (Cohan and Napelenok, 2011) in a commonly used CTM. The structure of this study is as follows: Sect. 2 outlines the data source and methodology utilized. Section 3 describes the current status of O₃ simulation studies in China and proposes recommended evaluation metrics and associated benchmarks. Section 4 delves into a discussion on O₃ uncertainties arising from different model inputs, and conclusions are given in Sect. 5.

2 Methodology

2.1 Data collection

The methodology for data compilation was consistent with our prior studies for other criteria pollutants (Huang et al., 2021; Zhai et al., 2024) and is briefly described here. We considered published O₃ simulations using five CTMs: the Community Multiscale Air Quality (CMAQ, https://www.epa.gov/cmaq, last access: 12 July 2024) model, the Comprehensive Air Quality Model with extensions (CAMx, https://camx.com, last access: 12 July 2024), GEOS-Chem (https://geoschem.github.io, last access: 12 July 2024), the Weather Research and Forecasting model coupled with Chemistry (WRF-Chem, https://www2.acom.ucar.edu/wrf-chem, last access: 12 July 2024), and the Nested Air Quality Prediction Modeling System (NAQPMS) (Wang et al., 2014; Ge et al., 2014). We gathered relevant publications using a combination of three keywords in the Web of Science, namely O₃, the models' names (one of the five models), and China, for studies published between 2006 and 2021. This process identified a total of 667 records (250 studies for CMAQ, 186 for WRF-Chem, 163 for GEOS-Chem, 36 for CAMx, and 32 for NAQPMS), with subsequent refinement steps to exclude duplicates, non-English publications, conference papers, and journals unrelated to air quality. Through manual selection, which involved identifying studies that provide extractable results (i.e., studies offering explicit results from model performance evaluations), a final set of 216 studies was chosen for detailed analysis (see Table S2 for a complete list of publications).

Figure 1 CMAQ modeling domain with definitions of regions used in this study. The surrounding pie charts display the total number of studies for each region (excluding studies for the whole of China) and the percentage of different CTMs used. Red stars represent the five cities selected for the uncertainty analysis. Publisher's remark: please note that the above figure contains disputed territories.

Different configurations could be used, even within the same model. For example, WRF-Chem provides different chemical mechanisms, ranging from the simple RADM2 without aerosols to the MOZART chemical mechanism with hundreds of species. Detailed information regarding model configurations (e.g., modeling period, horizontal resolution, gas-phase chemistry, initial and/or boundary conditions) and the results of 23 MPE metrics (Table S3) were extracted and compiled from those 216 studies. For consistency, we converted O₃ concentrations (for example, mean bias, root mean square error) expressed in parts per billion by volume (ppbv) to µg m⁻³ using a factor of 2.14. This factor of 2.14 refers to the “standard state”, i.e., an ambient air temperature of 273.15 K at 101.325 kPa, defined by the Chinese ambient air quality standards (GB 3095-2012; Ministry of Ecology and Environmental of the People's Republic of China, 2016). A total of 10 regions in China (Table S4), including the BTH region, the YRD region, the Pearl River Delta (PRD) region, the Sichuan Basin (SCB), the North China Plain (NCP), and five other regions (Fig. 1), were identified for further analysis.

2.2 Recommended benchmarks for O₃

Among the 23 collected MPE metrics, we derived recommended benchmarks for the six most frequently used metrics (see Table S5 for definitions): mean bias (MB), normalized mean bias (NMB), root mean square error (RMSE), normalized mean error (NME), correlation coefficient (R), and index of agreement (IOA). The derivation of benchmarks follows previous studies by Simon et al. (2012) and Emery et al. (2017). Briefly, each metric's rank-ordered (from best to worst, for instance, from 1 to 0 for R) distribution was generated to identify the values at the 33rd and 67th percentiles. As highlighted in Emery et al. (2017), these percentiles serve to categorize the entire distribution into three performance categories: studies falling within the 33rd percentile (the goal) attain the best performance that current models can be expected to achieve, those between the 33rd and 67th percentiles (the “criteria”) attain the typical performance achieved by the majority of modeling studies, while those beyond the 67th percentile indicate relatively poor performance for the particular metric under consideration. We present the benchmarks for hourly O₃, maximum daily 8 h average O₃ (8 h max O₃), and daily maximum 1 h O₃ (1 h max O₃), depending on data availability.

2.3 Uncertainty analysis of O₃ simulation

In addition to developing the MPE benchmarks for simulated ozone, we further quantified uncertainties in predicted ozone concentrations using one of the five models (i.e., CMAQ). CMAQ version 5.3.2 (https://www.epa.gov/cmaq) was employed to simulate O₃ during June 2021 in China. Base model configurations are the same as our previous study (Sun et al., 2024) and are briefly described here. The modeling domain covers the entirety of China and adjacent Asian regions (Fig. 1), with a horizontal resolution of 36 km × 36 km and 23 vertical layers and with a top pressure of 10 hPa. Meteorological fields are simulated using the Weather Research and Forecasting model (WRF version 4.0), the model configurations of which are listed in Table S6. CB6 and AERO7 were chosen as the gas-phase and aerosol mechanisms, respectively. Emission data include the 2019 Multi-resolution Emission Inventory for China (MEIC-2019) (http://www.meicmodel.org, last access: 23 June 2022) and the 2010 Emissions Database for Global Atmospheric Research (EDGAR, https://edgar.jrc.ec.europa.eu/, last access: 23 June 2022). Natural emissions were generated based on the Model of Emissions of Gases and Aerosols from Nature (MEGAN version 3.1, https://bai.ess.uci.edu/megan, last access: 23 June 2022). The CMAQ default O₃ profile (with a uniform O₃ concentration of 29 ppb) was used as the initial and boundary conditions (BCs). The use of a spatially and temporally uniform ozone concentration is a rather simplistic assumption, and, as we illustrate later, the impact of boundary conditions within the domain can range from substantial to minimally impactful. Among the CMAQ application studies collected, 54 out of 90 describe the configuration of the initial and boundary conditions, and 35 of those applied the CMAQ default profile. Since our purpose in the ozone uncertainty analysis was to quantify how variability in boundary conditions affects simulated ozone concentrations throughout China, we elected to mirror how many of the studies have applied CMAQ. A 10 d spin-up run was conducted to mitigate the influence of initial conditions.

We followed Dunker et al. (2020) to quantify the uncertainties in predicted O₃ concentrations due to six model inputs: anthropogenic NO_x (ANO_x) and VOC (AVOC) emissions for China, biogenic VOCs (BVOCs) and soil NO_x (SNO_x) within China, dry deposition velocities for O₃, and BCs for O₃. The uncertainties associated with each of the inputs (Table S7) are based on previous studies addressing emission uncertainties (Cheng et al., 2019), deposition velocities, and BCs (Beddows et al., 2017; Derwent et al., 2018). Like Dunker et al. (2020), these uncertainties were considered to be independent and log-normally distributed. The CMAQ decoupled direct method (DDM) was used to generate the first-order sensitivities of O₃ to each of the inputs (excluding dry deposition). For dry deposition, we conducted two parallel simulations in which the O₃ dry-deposition velocities were manually changed by ±10 %, and the changes in simulated O₃ concentrations were treated as the O₃ sensitivities to dry-deposition velocity:

$$\begin{array}{}\text{(1)}& S_{\mathrm{DEP}}^{\left(\mathrm{1}\right)}={\displaystyle \frac{C_{\mathrm{1.1}\mathrm{dep}\mathit{\_}{\mathrm{O}}_{\mathrm{3}}}-C_{\mathrm{0.9}\mathrm{dep}\mathit{\_}{\mathrm{O}}_{\mathrm{3}}}}{\mathrm{2}}}\times \mathrm{10},\end{array}$$

where $S_{\mathrm{DEP}}^{\left(\mathrm{1}\right)}$ is the O₃ sensitivity to dry-deposition velocities, and $C_{\mathrm{1.1}\mathrm{dep}\mathit{\_}{\mathrm{O}}_{\mathrm{3}}}$ and $C_{\mathrm{0.9}\mathrm{dep}\mathit{\_}{\mathrm{O}}_{\mathrm{3}}}$ represent the simulated O₃ concentrations as dry-deposition velocities are increased and decreased by 10 %, respectively. The sensitivities obtained were then combined with their respective uncertainties, enabling us to quantify the contributions to the variance in O₃ concentrations. For example, the O₃ uncertainties due to dry deposition are calculated as follows:

$$\begin{array}{}\text{(2)}& \mathrm{un}\left(\mathrm{DEP}\right)=\mathrm{var}\left(\mathrm{DEP}\right)={\left[{\displaystyle \frac{\ln \left(f_{\mathrm{DEP}}\right)}{\mathrm{2}}}\times S_{\mathrm{DEP}}^{\left(\mathrm{1}\right)}\right]}^{\mathrm{2}},\end{array}$$

where un(DEP) represents the uncertainty of O₃ due to dry deposition at 1σ, and f_DEP (equalling 2 based on Table S7) is the uncertainty factor for dry deposition and follows the assumption of a log-normal distribution.

The contribution of dry deposition to the total uncertainty in O₃ is calculated as follows:

$$\begin{array}{}\text{(3)}& \mathit{\%}\mathrm{DEP}={\displaystyle \frac{\mathrm{var}\left(\mathrm{DEP}\right)}{\begin{array}{c}\mathrm{var}\left({\mathrm{ANO}}_x\right)+\mathrm{var}\left(\mathrm{AVOCs}\right)+\mathrm{var}\left({\mathrm{BNO}}_x\right)\\ +\mathrm{var}\left(\mathrm{BVOCs}\right)+\mathrm{var}\left(\mathrm{DEP}\right)+\mathrm{var}\left(\mathrm{BCs}\right)\end{array}}}.\end{array}$$

3 Results and discussion

3.1 General overview of O₃ simulation studies in China

In the last decade, there has been a significant increase in research focusing on O₃ in China, as illustrated in Fig. 2. The issuance of the Three-Year Action Plan for Winning the Blue Sky Defense Battle in 2017 (http://www.gov.cn/zhengce/content/2018-07/03/content_5303158.htm, last access: 15 April 2024) led to a further surge in studies related to O₃, with a noticeable decline in 2020, which is possibly attributable to the impact of the COVID-19 pandemic. In 2021, there were 48 studies dedicated to addressing O₃-related issues using CTMs, marking a 6-fold increase compared to 2011. Similarly to PM_2.5, the BTH (74 studies), YRD (59 studies), and PRD (58 studies) regions emerged as the top three most studied regions. Among the various CTMs employed, CMAQ stood out as the most commonly utilized model (90 studies), followed by WRF-Chem (84 studies). The application of CAMx (14 studies) and NAQPMS (8 studies) was less frequent by comparison. In terms of MPE metrics, R had the highest frequency of occurrence at 19 %, followed by NMB (18 %), MB (16 %), RMSE (13 %), and NME (11 %). Nearly half of the studies incorporated two or three metrics for evaluating O₃, while less than 7 % assessed at least five different metrics. The three most common types of O₃ concentrations evaluated were hourly O₃ concentration, the maximum daily 8 h average O₃ (8 h max O₃), and the daily maximum 1 h O₃ (1 h max O₃). Among all the articles examined, 77 % focused on evaluating hourly O₃, 16 % focused on 8 h max O₃, and 7 % focused on 1 h max O₃.

Figure 2 Number of O₃ studies published during 2006–2021. Pie charts show the frequency of different MPE metrics (left) and the number of metrics used in one study (right).

3.2 Quantile distributions of O₃ MPE results

Figure 3 shows the quantile distributions of various evaluation metrics collected in this study. The results are presented for different types of O₃ concentrations, namely hourly O₃, 1 h max O₃, and 8 h max O₃, whenever data are available. Previous studies have shown that using maximum O₃ values (i.e., 1 h max and 8 h max) instead of hourly O₃ can lead to differing results within the same study (e.g., Ni et al., 2020; Li et al., 2016). Peak O₃ concentrations typically occur between 12:00 and 18:00 LT (Beijing Time, UTC+8). For example, in Ni et al. (2018), 8 h max O₃ showed an overestimation tendency compared to average hourly O₃, but, in another study (Yang et al., 2021b), there was an opposite trend. Underestimation of peak O₃ concentrations might be offset by overestimation during non-peak hours and vice versa. Therefore, achieving satisfactory performance in daily averaged O₃ levels does not necessarily indicate the model's ability to accurately capture high O₃ concentrations.

Figure 3 Quantile distribution of common O₃ performance indicators.

Hourly O₃ exhibited equivalent overestimation and underestimation in terms of MB and NMB, with MB ranging from as low as −40 to nearly 50 µg m⁻³ and with NMB ranging from less than −50 % to more than 70 %. However, fractional bias (FB) indicated more underestimated than overestimated hourly O₃ concentrations. For all three bias metrics, 8 h max O₃ exhibited more overestimation than underestimation, suggesting a tendency for models to overestimate off-peak hours. For 1 h max O₃, both NMB and FB displayed equivalent overestimation and underestimation, with NM showing a wider range than FB, likely due to there being fewer data points. For error metrics, 8 h max and 1 h max O₃ generally performed better than hourly O₃. For instance, the median values of NME were 34.8 %, 26.6 %, and 29 % for hourly O₃, 8 h max, and 1 h max O₃, respectively. R and IOA indicate how well the model captures observed variations, either temporally or spatially. The use of IOA was significantly less than R, and no studies reported IOA values for 1 h max O₃. For the other two O₃ types, IOA values (median value of 0.8 for O₃ and 0.77 for 8 h max O₃) were generally higher than R (median value of 0.69 for O₃ and 0.66 for 8 h max O₃). Six studies reported both R and IOA values, of which four (Liu and Wang, 2020; Wang et al., 2019; Liu et al., 2019b; Gao et al., 2017) reported higher IOA values than R.

3.2.1 Regional and seasonal differences

Like our previous studies (Huang et al., 2021; Zhai et al., 2024), we discuss the influences of various key factors on model performance in simulating O₃ concentrations. We first considered whether there were discernible regional or seasonal differences. Figure 4 presents the distribution of R and NMB values, grouped by three key regions in China: the BTH, YRD, and PRD (see Table S4 and Fig. 1 for region definition). These regions are the most densely populated and economically developed urban clusters in China. In terms of hourly O₃, the R values across the three regions display similarity, with median values of around 0.7. For 8 h max O₃, however, the PRD stands out with notably lower R values compared to the BTH and YRD. Regarding NMB values, the BTH tends to have more underestimation, while the YRD and PRD lean towards overestimation. Over the past decade, the BTH has consistently recorded the highest O₃ levels and number of O₃ pollution days among the three regions (Wang et al., 2024). The variations in NMB values among regions suggest a trend of current models underestimating O₃ levels in areas with more severe O₃ pollution.

Figure 4 Quantile distribution of R and NMB of O₃ in the BTH, YRD, and PRD.

Figure 5 Quantile distribution of O₃ R and O₃ NMB values in different seasons.

In terms of the seasonal variations (Fig. 5), the NMB values of hourly O₃ concentrations exhibit similar patterns across different seasons, showing equivalent overestimation and underestimation. However, when assessed over the entire year, hourly O₃ concentrations tend to be largely underestimated. The seasonal patterns of NMB distributions are similar for 8 and 1 h max O₃, with summer O₃ concentrations being more frequently underestimated compared to in other seasons. For instance, in the case of 1 h max O₃, peak O₃ concentrations are predominantly underestimated (with a median NMB of −23 %), while they are overestimated in winter (with a median NMB of 31.5 %).

3.2.2 Impact of horizontal resolution

The selection of a horizontal resolution for a CTM application depends on several factors, such as the objective of the study, the geographical scope of the study area, and the availability of input data. Generally, a coarse horizontal resolution (> 50 km) is utilized for global simulations (i.e., GEOS-Chem applications at a global scale), while a finer horizontal resolution (< 4 km) with nested grids is preferred for regional- or city-scale modeling. A coarser horizontal resolution may result in multiple monitoring stations falling within a single grid cell, potentially smoothing out extreme values observed at specific locations. Among the 216 studies reviewed, 29 different horizontal resolutions (based on the resolution of the innermost domain) were identified, ranging from 1 to 200 km. The horizontal resolutions were classified into five groups in this study: < 5, 5–10, 10–25, 25–50, and 50–100 km (horizontal resolutions over 100 km were excluded from the analysis due to limited data points). Figure 6 shows the distribution of eight statistical indicators by different horizontal resolutions while ignoring the differences in other model configurations. Overall, no clear trend was evident to indicate better model performances as horizontal resolution decreases. For example, the median R value is 0.73 for the < 5 km group, surpassing the 5–10 and 25–50 km groups but falling below the 10–25 and 50–100 km groups. Studies conducted with a horizontal resolution of 10–25 km exhibit the best model performance in terms of NME and FE distributions compared to other groups. While most studies assess models within a single domain (usually the innermost domain with the finest horizontal resolution), a few studies have conducted multi-domain analyses, where finer horizontal resolutions generally have superior results compared to coarse horizontal resolutions. Liu et al. (2020b) used WRF-CMAQ to analyze O₃ prediction and health exposure at different horizontal resolutions (1, 4, 12, and 36 km). The results showed more than 20 % difference in premature mortality due to different model horizontal resolutions being used. Therefore, modelers should exercise caution and avoid optimism when configuring their model at finer resolutions as reducing the horizontal resolution does not necessarily lead to improved model performance if the input data resolution (i.e., horizontal resolution of the emissions) is insufficient for the model's resolution.

Figure 6 Quantile distribution of O₃ with respect to commonly used assessment indicators at different horizontal resolutions.

3.2.3 Choice of gas-phase chemical mechanism

Gas-phase chemical mechanisms play a crucial role in the accurate prediction of atmospheric composition using CTMs. Some of the commonly used mechanisms include the carbon bond mechanism (CB) (Yarwood et al., 2010; Luecken et al., 2019; Appel et al., 2021; Yarwood and Tuite, 2024), the Statewide Air Pollution Researcher Center (SAPRC) mechanism (Carter, 1996; Chang et al., 1999; Carter, 2000, 2010), and the Regional Atmospheric Chemistry Mechanism (RACM) (Stockwell et al., 1997; Goliff et al., 2013). These mechanisms have undergone rigorous evaluations against experimental data, showcasing reliable predictive capabilities for O₃ in diverse atmospheric environments. The CB mechanism is a condensed mechanism in which the carbon bond is treated as a reaction unit, and the carbon bonds with the same bonding state are treated as a group (Cao et al., 2021). The latest version, CB7, contains 91 gaseous species and 230 reactions (https://www.tceq.texas.gov/downloads/air-quality/research/reports/photochemical, last access: 18 June 2024). In contrast, the SAPRC mechanism categorizes species based on their reactivity with OH (Carter, 2010). The most recent SAPRC22 mechanism includes 162 species and 738 reactions (https://intra.engr.ucr.edu/~carter/SAPRC/22/, last access: 18 June 2024). RACM was developed based on the Regional Acid Deposition Model (RADM), which is an inductive mechanism for treating hydrocarbons with a fixed parameterization method and is carried out according to the reaction rate and activity of different pollutants with •OH. Compared to the other two mechanisms, RACM and RACM2 contain detailed chemical processes of radicals, biogenic VOCs, and less reactive VOCs able to survive during long-distance transport. A total of 119 reactive species and 363 reactions were included in RACM2, describing the oxidation reactions of 21 types of primary VOCs in the system (Liu et al., 2023a).

Figure 7 Quantile distributions of R and NMB by gas-phase chemical mechanism.

Among the 216 studies compiled, nearly half of them used the CB mechanism for simulations, approximately a quarter employed RACM and/or RADM, and only 15 studies utilized SAPRC. Figure 7 compares the distribution of R and NMB, grouped by different gas-phase mechanism. In terms of R values, CB tends to perform slightly better than RACM and/or RADM, with SAPRC showing the highest R median value (0.93) for hourly O₃ but the lowest for 8 h max O₃ among the three mechanisms. Regarding NMB, SAPRC tends to overestimate peak O₃ values compared to the other mechanisms, particularly for 1 h max O₃, a trend observed in previous studies (Qiao et al., 2019).

3.3 Recommended benchmarks for O₃ MPE

Figure 8 illustrates the ranked distributions of various statistical indicators, including R, IOA, NMB, NME, FB, and FE for hourly O₃, 1 h max O₃, and 8 h max O₃. The absolute values of NMB and FB are presented to indicate deviations from zero. In terms of R and IOA, the ranked distributions for hourly O₃ and 8 h max O₃ are quite similar, with R values ranging from around 0.72 at the 33rd percentile to 0.60 at the 67th percentile. The corresponding IOA values are slightly higher, ranging from ∼ 0.83 at the 33rd percentile to ∼ 0.73 at the 67th percentile. For 1 h max O₃, the limited number of data points (less than 20) resulted in an R value of 0.80 at the 33rd percentile and 0.60 at the 67th percentile, while the IOA distribution was not available due to missing data. For NMB and NME, the results for 8 h max O₃ show the lowest values, indicating that models perform better in capturing the 8 h max O₃ concentrations. The 33rd percentile of absolute NMB for 8 h max O₃ is less than 10 %, and the 67th percentile is below 20 %. In terms of FB and FE, the ranked distributions for 1 h max O₃ are flatter compared to the other two O₃ types, likely due to the smaller number of available data points. For both metrics, the 8 h max O₃ exhibits lower values than O₃. At the 33rd percentile, the absolute FB (FE) is less than 10 % (25 %) for 8 h max O₃ and less than 20 % (50 %) for O₃. At the 67th percentile, the absolute FB (FE) is 25 % (38 %) for 8 h max O₃ and 34 % (65 %) for O₃. In addition, we provide a more detailed ranked distribution in Table S8.

Figure 8 Rank-ordered distributions of R, IOA, NMB, NME, FB, and FE for O₃, 1 h max O₃, and 8 h max O₃ speciated components. The number of data points and the 33rd, 50th, and 67th percentile values are also listed.

Table 1 Recommended benchmarks for evaluating simulated O₃ by CTM application in China. NA: not available.

(1) See descriptions in the main text for bold values​​​​​​​; (2) values with an asterisk indicate that our benchmarks are stricter than the corresponding values in Emery et al. (2017).

Following Emery et al. (2017) and Huang et al. (2021), we propose recommended statistical indicators and corresponding benchmarks for evaluating O₃, as detailed in Table 1. The goal values, corresponding to the threshold at the 33rd percentile, represent the optimal model performance anticipated from current models. The criteria values, reflecting the threshold at the 67th percentile, represent the performance levels achieved by the majority of studies. Due to limited data availability, the derivation of benchmarks for certain metrics concerning 1 h max O₃ remains uncertain. In such cases, benchmarks for IOA and R for hourly O₃ were directly adopted due to minimal variations among different O₃ types. Similarly, benchmarks proposed for 8 h max O₃ were applied to 1 h max O₃ for FB and FE, given their closer distributions. Our findings indicate that benchmarks tend to be more stringent for 8 h max O₃ compared to for the other two types, with the exception of IOA, where they remain the same. Based on our results, a value of R greater than 0.70 and 0.55 would meet the goal and criteria benchmarks, respectively, for 8 h max O₃. Correspondingly, the goal and criteria values for NMB are 10 % and 20 %.

In contrast to Emery et al. (2017), we provide separate benchmarks for O₃, 8 h max O₃, and 1 h max O₃. Emery et al. (2017) found rather similar results between hourly and 8 h max O₃ in the US and so recommended a single set of benchmarks for ozone. Out of the 216 studies analyzed, 15 studies evaluated at least two O₃ types. The use of a cutoff for evaluating O₃ is extremely limited in China (only five studies applied cutoffs), thereby precluding any specific recommendation regarding cutoff values. In addition to the benchmarks for NMB, NME, and R provided by Emery et al. (2017), we have introduced benchmarks for IOA, FB, and FE, backed by a sufficient number of data points. The few values marked with an asterisk in Table 1 indicate that our benchmarks are more stringent than the corresponding values in Emery et al. (2017), implying that achieving our recommended 33rd (or 67th) percentile may pose greater challenges.

Overall, however, our proposed benchmarks are more lenient than those of Emery et al. (2017), particularly in the context of hourly O₃. For NME, our suggested goal and criteria for O₃ stand at 30 % and 45 %, respectively, nearly double the figures reported by Emery et al. (2017), who recommend 15 % for the goal and 25 % for the criteria. The criteria value for R is an exception, where our proposed value (0.55 for 8 h max O₃ and 0.60 for O₃) is higher than that (0.50) in Emery et al. (2017).

3.4 Uncertainty analysis of O₃ simulation using CMAQ

In order to further investigate the uncertainties in O₃ concentrations simulated by CTMs, a base model simulation was conducted using CMAQ (the most frequently used CTM in China) for June 2021, a typical month with elevated O₃ in northern and eastern China. The uncertainties due to six model inputs were quantified for this case: VOC and NO_x emissions in China, differentiation between anthropogenic and biogenic sources, O₃ dry-deposition velocities, and boundary conditions (BCs). The evaluation of the base model results indicates generally acceptable simulated MDA8 O₃ concentrations when compared to the observations. The results showed an overall MB of 6.1 µg m⁻³ and an overall NMB of 5.2 % (Fig. 9). O₃ underestimation is observed over the BTH region, while overestimation occurs over the Sichuan Basin. The values of NMB, NME, and R meet the goal benchmark we proposed above.

Figure 9 Spatial distributions of (a) MDA8 O₃ concentrations (µg m⁻³), (b) total uncertainties (µg m⁻³), and (c) total uncertainty in percent (%). Results are averaged for June 2021. Publisher's remark: please note that the above figure contains disputed territories.

Figure 10 Contributions to uncertainty in MDA8 O₃ simulation. Contribution of (a) AVOCs, (b) BVOCs, (c) ANO_x, (d) SNO_x, (e) O₃ BCs, and (f) dry deposition in µg m⁻³. Results are averages over all days in June 2021 and represent 1σ. Publisher's remark: please note that the above figure contains disputed territories.

As displayed in Fig. 10, the first-order sensitivity of MDA8 O₃ to the six model inputs exhibits substantial variations in terms of spatial distributions and magnitudes. A higher sensitivity occurs in larger urban areas, while the sensitivity is relatively low in rural areas. The sensitivity to VOC emissions is always positive (i.e., higher VOCs lead to higher O₃), whereas the sensitivity to NO_x emissions could be both positive and negative. High O₃ sensitivity to AVOC emissions is observed for the BTH, the northern YRD, the PRD, and major metropolitan areas (e.g., Chengdu in Sichuan Province, Xi'an in Shaanxi Province) due to NO_x-rich and VOC-limited urban conditions. Conversely, anthropogenic NO_x emissions resulted in negative O₃ sensitivity in the aforementioned regions and positive sensitivity in others where rural conditions are more VOC-rich and NO_x-limited. The sensitivity to biogenic precursor emissions (BVOCs and SNO_x) was much lower compared to their anthropogenic counterparts. The sensitivity to O₃ BCs predominantly extends towards the northwest (up to 50 µg m⁻³), where O₃ precursor emissions are low. The sensitivity to O₃ dry-deposition velocity exhibits a uniformly negative distribution (higher deposition rates lead to lower ozone), with higher values in more vegetated areas and an average of −13.7 µg m⁻³.

When the individual first-order sensitivity coefficient is multiplied by the corresponding 1σ uncertainty (Table S7), the contributions to the uncertainty in O₃ predictions can be obtained (Fig. 10). Summing up all these uncertainties yields the total uncertainty (Fig. 9b). Large ozone uncertainties (> 20 µg m⁻³) were observed over the BTH, the central YRD region, and major metropolitan areas (e.g., the PRD, Chengdu in Sichuan Province). Regions with high uncertainties in O₃ predictions generally align with regions with poorer model performance. In the BTH, YRD, and PRD, the total ozone uncertainty due to the six model inputs is between 11.7–31.8, 7.0–34.6, and 5.0–19.0 µg m⁻³, respectively, corresponding to a relative percentage of O₃ concentration of 9.2 %–18.1 %, 7.9 %–25.8 %, and 7.6 %–14.6 %. It should be noted that our uncertainty estimates represent conservative estimates because the effects of uncertainties in the meteorological inputs and the uncertainties associated with the O₃ chemistry are not included, the latter of which have been shown to have a comparable contribution to the total contributions from emissions, dry deposition, and O₃ BC in the Dallas–Fort Worth region in the US (Dunker et al., 2020).

Among the six model inputs, AVOC emissions make the largest contributions (exceeding 15 µg m⁻³) to the total uncertainty in regions displaying high O₃ sensitivity, such as the BTH, the northern YRD, the PRD, and several metropolitan areas. The large uncertainties, stemming from both the high first-order sensitivities (Fig. S1) and a relatively high uncertainty factor (1.97), suggest that, in these regions, uncertainties associated with AVOC emission estimates would result in more significant biases in simulated O₃ concentrations compared to in other areas. O₃ uncertainties due to BVOC emissions, ranging between 0.1–10.4 µg m⁻³, are mainly located in southern China, where BVOC emissions are high. A similar spatial pattern is observed for uncertainties in ANO_x emissions, although its contribution is larger (0.5–11.9 µg m⁻³). While the first-order O₃ sensitivity to SNO_x emissions is minimal (Fig. S1), the contribution to O₃ uncertainty from SNO_x emissions is noteworthy (0.5–9.7 µg m⁻³), given a large uncertainty factor of 2 (Table S7). Uncertainty in O₃ BCs is relatively less important, except in the northwest, where it represents the largest contributing factor. Dry deposition serves as an important O₃ sink. The uncertainty contribution of O₃ dry-deposition velocities (0.3–10.4 µg m⁻³) is comparable to that of ANO_x emissions but has a more evenly distributed spatial impact.

Figure 11 Time series of simulated and observed MDA8O₃ for five cities in June 2021. The uncertainty limit of MDA8 O₃ is ±1σ. The pie chart shows the contribution of each factor to the total uncertainty of the predicted average MDA8 O₃ in June 2021.

Figure 11 compares the observed MDA8 O₃ to the model results, including their ±1σ uncertainty range for five major cities: Beijing, Shanghai, Guangzhou, Chengdu, and Xi'an. In Shanghai, the majority of the observed O₃ falls within the ±1σ uncertainty range. However, in Beijing; Chengdu; and, to a lesser extent, in Guangzhou, the model tends to overpredict lower O₃ observations. In Xi'an, the model fails to capture the exceptionally high O₃ concentrations (MDA8 O₃ > 250 µg m⁻³) on 6 and 7 June. Expanding the uncertainty limits to a ±2σ range may encompass some of the lower O₃ observations, but the current uncertainty estimates do not fully account for all the discrepancies between model results and observations. This discrepancy could be attributed to the coarse horizontal resolution (36 km) used in this study, which may not adequately resolve the impact of local emission sources. Furthermore, as mentioned earlier, uncertainties related to O₃ chemistry and meteorological inputs were not accounted for and should be quantified in future work.

The relative contributions to the total uncertainty are also shown in Fig. 11. Across all five cities, uncertainties in the AVOC emissions contribute the most (43 %–65 %), while the relative importance of other model inputs differs by location. For example, O₃ BCs represent the second largest uncertainty source in Beijing (accounting for 18 %) but are negligible in Guangzhou and Chengdu. In Shanghai and Guangzhou, uncertainties in ANO_x emissions (10 %–17 %) become the second largest contributor. Uncertainties associated with BVOC emissions are minimal in Beijing and Shanghai but are noteworthy (7 %–8 %) in Guangzhou and Chengdu. O₃ deposition uncertainty contributes to 8 %–30 % of the total uncertainty, with a higher contribution for cities located in the west.

3.5 Recommendations for future modeling practices

The purpose of this and our previous related papers (Huang et al., 2021; Zhai et al., 2024) is to establish a guideline that offers modelers in China a contextual reference for evaluating their statistical performance metrics against a historical framework of published modeling results. It is crucial to recognize that all models inherently possess a certain degree of error, which may arise from factors such as discretization, approximations, and parameterizations. Merely stating that a model exhibits a 20 % bias has no real value unless contextualized within the framework of historical performance. Without such context, it remains unclear whether this bias aligns with, surpasses, or falls short of commonly achieved standards or optimal expectations. Statistical results outside the proposed benchmarks indicate poor model performance that should be improved upon so that more reliance can be placed on the model to properly characterize air quality and predict responses to changes in model inputs. Based on the above analysis, we list several recommendations for future modeling practices that might help improve model performance.

1. Meteorology is an essential input to CTMs. Many studies have highlighted the strong responses of ozone to various meteorological variables (Coleman et al., 2013; Lu et al., 2019), including transport patterns, temperature, planetary boundary height, and relative humidity. Consequently, it is imperative to conduct a thorough validation of meteorological simulations prior to initiating ozone simulations. The influence of uncertainties associated with simulated meteorological variables on ozone predictions necessitates further exploration.

2. Modelers are encouraged to select the highest feasible horizontal resolution that matches the available emission data horizontal resolution. Our analysis illustrates that finer horizontal resolutions do not invariably lead to enhanced model performance, particularly when the input data do not possess a correspondingly high horizontal resolution. Thus, it is advisable for modelers to engage in sensitivity testing to ascertain the optimal equilibrium between horizontal resolution and data quality.

3. The uncertainty analysis reveals a substantial contribution of AVOC emissions throughout China. Therefore, it is essential to intensify efforts aimed at enhancing the accuracy of AVOC emissions, focusing on both magnitude and speciation profiles. Additionally, the chemical mechanisms within CTMs should be routinely updated to accommodate emerging species, such as volatile chemical products (VCPs; Yarwood and Tuite, 2024).

4. The majority of model applications reviewed in this study apply a spin-up period of less than or equal to 10 d. However, studies (Hogrefe et al., 2017; Karamchandani et al., 2017) have shown that a commonly used spin-up period of 10 d (or a week) might not be sufficient to reduce the effects of initial conditions to less than 1 %. Thus, a longer spin-up period, preferably 20 d, depending on the domain size, is recommended to mitigate the influence of initial conditions.

5. Given the considerable effect of boundary conditions on simulated ozone uncertainties – especially in areas characterized by low precursor emissions – modelers should carefully select and validate boundary conditions. This may involve using multiple global models or observational data to define more accurate initial and boundary conditions.

6. In the context of ozone attainment demonstrations, modelers should place a particular emphasis on the model's performance in relation to high and peak ozone values. Merely achieving satisfactory average ozone concentrations may not suffice; it is essential to ensure robust performance in capturing peak ozone levels as well.

4 Conclusions

Chemical transport models are increasingly being employed to tackle the severe ozone pollution issues in China. This study involved the compilation and analysis of 216 peer-reviewed studies focused on the use of CTMs to simulate O₃ levels in China. Essential model configurations such as study region, simulation season, horizontal resolution, gas-phase mechanism, and quantitative model performance outcomes were systematically documented. The study presented quantile distributions of common statistical metrics found in the literature and discussed the influence of different model configurations on performance outcomes. Furthermore, we proposed benchmarks for six widely used MPE metrics (R, IOA, NMB, NME, FB, and FE) based on the concepts of goals and standards to offer guidance to modelers for a more consistent and contextual evaluation of models. Additionally, we utilized CMAQ-DDM to assess the uncertainties in predicted O₃ concentrations resulting from uncertainties in six model inputs. The findings revealed significant variations in the spatial distributions and magnitudes of ozone sensitivity to different model inputs, with the most substantial contributions to total uncertainty originating from AVOC emissions in regions with high ozone sensitivity.

The proposed benchmarks for assessing simulated O₃ concentrations, in conjunction with previous studies on PM_2.5 (Huang et al., 2021) and other criteria air pollutants (Zhai et al., 2024), represent a comprehensive and systematic effort to establish a model performance framework for CTM applications in China. These outcomes not only offer valuable guidance to the growing modeling community in China but also support the community's endeavors in utilizing CTMs to address various research challenges and to enhance air quality management.