Detecting a tropospheric ozone trend from sparsely sampled ozonesonde profiles (typically once per week) is challenging due to the short-lived anomalies in the time series resulting from ozone's high temporal variability. To enhance trend detection, we have developed a sophisticated statistical approach that utilizes a geoadditive model to assess ozone variability across a time series of vertical profiles. Treating the profile time series as a set of individual time series on discrete pressure surfaces, a class of smoothing spline ANOVA (analysis of variance) models is used for the purpose of jointly modeling multiple correlated time series (on separate pressure surfaces) by their associated seasonal and interannual variabilities. This integrated fit method filters out the unstructured variation through a statistical regularization (i.e., a roughness penalty) by taking advantage of the additional correlated data points available on the pressure surfaces above and below the surface of interest. We have applied this technique to the trend analysis of the vertically correlated time series of tropospheric ozone observations from (1) IAGOS (In-service Aircraft for a Global Observing System) commercial aircraft profiles above Europe and China throughout 1994–2017 and (2) NOAA GML's (Global Monitoring Laboratory) ozonesonde records at Hilo, Hawaii, (1982–2018) and Trinidad Head, California (1998–2018). We illustrate the ability of this technique to detect a consistent trend estimate and its effectiveness in reducing the associated uncertainty in the profile data due to the low sampling frequency. We also conducted a sensitivity analysis of frequent IAGOS profiles above Europe (approximately 120 profiles per month) to determine how many profiles in a month are required for reliable long-term trend detection. When ignoring the vertical correlation, we found that a typical sampling strategy (i.e. four profiles per month) might result in 7 % of sampled trends falling outside the

The vertical profile is a type of time series data that reports the composition (e.g., ozone, water vapor, carbon monoxide) or thermodynamic properties (e.g., temperature, relative humidity, wind speed) of the atmosphere from the surface to an altitude that can range from a few tens of meters (e.g., tethered weather balloons) to more than 30 km (e.g., ozonesondes). Conventionally, the analysis of trends based on vertical profile data is conducted on particular altitude bins or pressure levels which are treated as independent time series (e.g.,

In terms of producing an area average based on multiple stations dispersed across a given region, a typical procedure is to create a surface gridded product, which usually aggregates all available monitoring stations within the grid cell without considering spatial sampling and irregularities

The aim of such aggregations, either through averaging or dimension reduction, is to enhance a certain common underlying signal (or signals) or to produce regional means

Our method uses a two-dimensional geoadditive model for carrying out the multiple correlated time series analyses on every level of the vertical profiles. This approach borrows the strength of spatial correlation models, but instead of working on dimensions of latitude and longitude, we replace the geographical coordinates with a temporal index and altitude bins. This approach also enables us to identify the common structures among correlated time series (e.g., ozone observations on adjacent layers of the atmosphere) through their seasonality and interannual variability and to filter out unstructured variations from the underlying signal by a regularization technique. We demonstrate that this method reduces the short-lived anomalies in an irregular and/or sparse time series by taking advantage of the additional information in adjacent layers of the vertical profiles and yields a trend with reduced uncertainty. While our method was developed for tropospheric ozone profiles, it can also be applied to the stratosphere, to profiles of other trace gases or atmospheric thermodynamic properties, and to profiles of oceanic or geologic properties.

Section 2 begins with a brief introduction of the time series decomposition and smoothing spline ANOVA framework. We then propose a statistical methodology to assess the profile variability. Section 3 applies this methodology to two tropospheric ozone profile datasets measured by (1) IAGOS (In-service Aircraft for a Global Observing System) commercial aircraft above Europe and China (including a probability test of how the trend precision and accuracy change according to sampling frequency) and (2) ozonesondes launched from Hilo, Hawaii, and Trinidad Head, California, by NOAA's Global Monitoring Laboratory (GML). In Sect. 4, we discuss the potential extension and benefit of this trend technique and provide a summary of our ozone profile trend analysis.

Conventional long-term trend detection of a time series is influenced by many factors such as the number of years of data, the magnitude of the uncertainty and autocorrelation of the residuals

Ozone variations are in general strongly dependent on the altitude or pressure level. For a given altitude or pressure level, our method borrows the information from the neighboring altitudes at approximately the same time for a more stable trend estimate (e.g., observations at nearby altitudes are more highly correlated with the layer of interest than those further away). In order to explicitly account for common structures among correlated time series, a class of generalized additive models (GAMs;

With an extension of a single time series to a vertical profile series (let

The two-dimensional penalized regression splines were chosen for representing both smooth components

The specification of our model enables us to decompose the vertical profile into two geoadditive fields: seasonal and interannual components, according to their associated variabilities. These surfaces are defined by approximating the overall monthly means and annual mean anomalies in each layer or altitude through a statistical optimization. Rather than directly specifying a linear component into a multivariate linear regression model, our functional approach aims to partition monthly mean ozone profiles into components attributable to different sources of variation before any attempt at trend detection.
We choose the thin-plate smoothing spline as the spatial basis function because it is computationally efficient and because it avoids the problem of choosing “knot locations”

Our new trend calculation methodology is applied to the long-term tropospheric ozone vertical profile datasets measured by (1) IAGOS commercial aircraft above western Europe and China at 50 hPa intervals from 950 to 250 hPa for the period 1994–2017 and (2) ozonesondes launched from Hilo, Hawaii, (1982–2018) and Trinidad Head, California, (1998–2018) with a vertical resolution of 100 m from the surface to 15 km. For consistency the vertical coordinate of the ozonesondes is converted from meters to pressure levels.

The IAGOS program has measured ozone worldwide since 1994, using instruments on board commercial aircraft of internationally operating airlines (

NOAA's Global Monitoring Laboratory has measured ozone profiles above Hilo, Hawaii, (19.72

The IAGOS dataset in western Europe is a regional aggregation of profiles from several airports. The profiles are measured by several aircraft, and on average there are approximately four profiles per day within the study region. Several studies have demonstrated that IAGOS data above western Europe are consistent with ozonesonde records in the upper troposphere–lower stratosphere (UTLS)

With such a high sampling frequency, the data are expected to be far less inconsistent than a sparsely sampled ozonesonde time series (one to three profiles per week) and highly representative of the true ozone variability above western Europe. The IAGOS dataset in China is also a regional aggregation, but the time series has a period of low sampling frequency (2007–2010) with many months containing no data. In contrast, the ozonesonde records at Hilo and Trinidad Head are expected to have greater uncertainty due to the lower sampling rate of one profile per week. Our first task was an evaluation of our method against the standard linear model using the IAGOS data above Europe to determine if our method yields trend values with reduced uncertainty. The next step was to examine the impact of sampling frequency on the trend estimate based on the IAGOS profiles above western Europe. Finally, we applied our method to the sparsely sampled Hilo and Trinidad Head ozonesonde records to demonstrate the improved trend quantification.

Figure

Correlation coefficients between different pressure layers based on IAGOS profiles above western Europe.

The smooth seasonal and interannual components in units of parts per billion by volume (ppbv) derived from Eq. (

Seasonal and interannual components for the ozone distribution above western Europe.

From a visual inspection of the interannual component, the variability in the lower layers is fairly steady and unwavering compared to the upper layers. The largest variation over 1994–2017 occurred in the upper layers around 2017 (the absolute magnitude of the variability during 2017 is still in question as it was the last year of the available time series and could be modified by additional years of data as they become available). The result from the smoothing spline decomposition provides a good visualization of the trend and also provides a quantification of ozone variation through the multiple correlated time series using all profile data without a dimension reduction.

Figure

Diagnosis of statistical model fitting of the tropospheric ozone distribution above western Europe.

Using the interannual component in Fig.

As shown in Fig.

Ozone trend estimates and associated

Monthly mean ozone time series and model fitted values for four different layers above western Europe.

To demonstrate the role of regularization in the model fitting, we first provide a synthetic example that illustrates the problems of underfitting (the model is not flexible enough to capture the general pattern in the data) and overfitting (the model is overparameterized, so some unphysical fluctuations are present) in Supplement Fig. S1. Neither underfitting nor overfitting is an appropriate representation of the true process. Once sufficient model complexity is supplied (e.g., placing a knot for spline functions every 50 hPa), the statistical regularization can be used to penalize overly complex models and thus prevent overfitting. The overfitting of a surface is less obvious than a curve, but we provide a demonstration of the fitting by adjusting the optimized penalty coefficient, which is selected by the generalized cross validation

Sensitivity analysis of the fitted result from different smoothing penalties:

Figure

Due to ozone's high temporal variability, accurate trend detection may not be possible if the ozone profile sampling rate is low. Three previous studies have estimated the optimal sampling frequency for the accurate quantification of tropospheric ozone variations from profiles.

Here we determine the optimal ozone profile sampling frequency for trends calculated using either the separated fit or the integrated fit methods. In order to investigate the impact of sampling frequency on profile trends, we randomly selected a specified number of profiles per month (from 1 to 20) and conducted the same integrated fit analysis on the resulting monthly mean time series based on 1000 iterations of random sampling. We illustrate the impact of the sampling frequency by showing the trend results based on one, five and nine profiles per month in Fig.

Sensitivity analysis for one

Sensitivity analysis at 550 hPa based on 1000 random samples of one

If we assume that the trend derived from the full dataset (dozens of profiles per month) is the “true” value, the sampled trend is randomly located around the true value as a Gaussian distribution for both approaches. For both the integrated and separated fit methods, the increased sampling frequency results in a diminished distribution of trends. The integrated fit is designed to smooth out the short-lived anomalies in the vertical profiles, and as a result the integrated fit produces a narrower range of trend distributions than the simple separated fit. This case study clearly illustrates that an inconsistent trend structure results from infrequent sampling.

The summary statistics for the sensitivity analysis in terms of both precision and accuracy are reported in Table

Percentage of randomly generated trends that fall outside of the 1 or

As an indication of the trend precision, we calculate the percentage of the 15 000 random trend values that fall outside of the

Figure

The marginal decrement of the outlier rate (sampled trends located outside the 1 or

In order to investigate the impact of sampling strategy on the integrated and separated fits, we carry out an additional sensitivity analysis in Supplement Table S2 through a comparison of two strategies: (a) a completely random design as illustrated above and (b) a fixed sampling strategy based on the random selection of an initial day followed by additional profiles at fixed intervals of 1 to 10 d. For example, a 5 d sampling frequency is based on the random selection of an initial reference from day 1 to day 5 in the beginning of the record, followed by the random selection of a profile every 5th day until the end of the record. For both strategies, only a single profile is chosen randomly if multiple profiles are available on the same day. Therefore, the sampling scheme with a fixed frequency of 1 d represents a random selection of a single profile in each day instead of using all available data. Also, if the chosen day does not have any profiles, we treat it as missing and do not look for a replacement.

The sensitivity analysis demonstrates that sample size remains the dominant influence on precision and accuracy. When the sampling interval is not dense enough (e.g., greater than 10 d), the benefit of a regular frequency scheme is inconsequential. However, when the monthly sample size is greater than four profiles (i.e., the sampling frequency is less than once per week), the fixed frequency scheme could achieve a better performance. As a result, an optimal frequency can decrease to 10 profiles (3 d frequency) for an integrated fit and 15 profiles (2 d frequency) for a separated fit.

The above discussion is focused on the precision and accuracy of the trend estimate at various sampling frequencies. In addition, we can explore the impact of sampling frequency on our ability to simply detect the presence of a trend based on uncertainty analysis. In order to evaluate the resulting uncertainty associated with the sampled trends, we include the mean signal-to-noise ratio (MSNR) between trends derived from the full dataset (the assumed true trend values) and the uncertainty in each sample (i.e., standard error associated with the trend estimate) in Table 1. In the interest of a fair comparison, this calculation is done by comparing the sampling uncertainty with trend values derived by the same method. Note that we do not use the concept of “statistical significance” to indicate evidence for the trends, following the recent recommendations from the American Statistical Association

IAGOS aircraft have measured ozone profiles above northeastern (NE) China and Seoul, South Korea, since 1994 but at a much lower sampling rate than western Europe. While western Europe has 36 298 profiles during 1994–2017, NE China (including profiles from Seoul, South Korea) only has 1636 profiles with many months containing sparse data during 2007–2010. The lower sampling frequency and the data gap mean that the NE China IAGOS dataset can provide a further test of the advantages of the integrated fit method. The seasonal and interannual components of the integrated fit to NE China are shown in Fig.

Seasonal and interannual components for the ozone distribution above NE China.

In this particular region, the variogram fitting suggests that ozone time series within a range of 200 hPa should be considered correlated. This vertical correlation range is greater than was found for the IAGOS data above western Europe. The longer correlation range is the result of the more systematic temporal variations across ozone vertical profiles, as seen in Fig.

The resulting trend distribution above China is displayed in Fig.

Ozone trend estimates and associated

Ozonesondes have been launched from Hilo, Hawaii, continuously since 1982 at an average rate of three profiles per month for the first 10 years and four profiles per month since 1993; the sampling rate at Trinidad Head has been weekly since August 1997. Figure

All ozone observations below 15 km as measured by ozonesondes above Hilo, Hawaii, and Trinidad Head, California. The observations are colored according to the decade in which they were measured. The solid lines represent the 5th, 50th and 95th percentiles. The vertical lines represent a reference at 10, 40 and 80 ppb.

The results of the functional decomposition to distinguish the seasonal and interannual components of the Hilo record are shown in Fig.

Seasonal and interannual components for the ozone distribution above Hilo, Hawaii.

Note that increasing the vertical resolution of the profile data does not play an important role in the quantification of the vertical correlation range as the correlation range of a fitted variogram is still approximately 150–200 hPa in the troposphere for these two stations, as previous results suggested. The vertical distributions of the Hilo (1982–2018) and Trinidad Head (1998–2018) ozone trends from the separated and integrated fits are shown in Fig.

Seasonal and interannual components for the ozone distribution above Trinidad Head, California.

For the trend distribution at Trinidad Head, the overall trend profile is shifted toward negative values due to the positive ozone anomaly in 1998–1999 (as described above), but above 900 hPa the

Ozone trend estimates and associated

Detecting trends of tropospheric ozone from ozonesonde profiles is challenging due to the relatively low sampling frequency combined with high temporal variability. Regularization is a statistical learning tool which makes a trade-off between (1) high fitted bias (low flexibility) that results from underfitting and (2) high sensitivity to small data fluctuations (low generalizability) that results from overfitting. The underfitting can be avoided by making sure the number of basis functions (e.g., thin-plate splines) is large enough to represent the underlying process. A model can be considered to be overfit if a high cross validation error is found (which can be investigated by, e.g., iteratively removing one observation and predicting this value from the remaining observations). In terms of detecting tropospheric ozone trends from vertical profiles, the vertical correlated structures in the neighboring pressure layers can be used to inform the learning process. The benefit of this approach can be reflected by the detectable trends (if any) at a low sampling rate (i.e., we have higher confidence of our ability to detect a trend from weekly sampled ozonesonde data) and by an improved quantification of trend estimates in terms of accuracy and precision.

This technique efficiently reduces the uncertainty of the resulting trends and thus increases our ability to detect and quantify trends of smaller magnitude. Therefore, this method is valuable for improved trend detection of ozonesonde records because, although these records are sufficiently long term and have a high vertical resolution with high accuracy, standard trend analysis is still challenging due to the limited sampling rate. This refined estimation is also expected to be beneficial when comparing trends derived from different regions or observing systems since the result is less sensitive to incoherent or unstructured variations.

This paper provides a sophisticated statistical regularization approach for carrying out a joint trend analysis of vertical profile data as applied to tropospheric ozone observations from IAGOS commercial aircraft profiles above Europe and China and NOAA GML's ozonesonde records at Hilo, Hawaii, and Trinidad Head, California. Our approach is designed to deliver a consistent trend and uncertainty estimate by functionally decomposing the overall vertical profile into seasonal and interannual components. Instead of fitting the trend component as a single linear term, our technique adopts a two-step data-driven approach that (1) expresses the continuous change as a series of estimated annual mean anomalies at each altitude or pressure level after accounting for de-seasonalization and vertical autocorrelation and (2) then derives the trend value from the annual mean anomalies according to its underlying structure (a linear approximation of the interannual variability is considered to be appropriate). In this study, we introduce roughness penalties on both monthly and annual scales, which are the fundamental elements of time series decomposition. This data-driven approach reveals the underlying trend structure without determining the linear, nonlinear or any other type of polynomial in advance. The success of this approach lies in the smoothing spline decomposition of ozone profile variability. Importantly, the regularization aspect diminishes the influence from extreme observations. Our method is easy to implement with low computational costs, and it offers a spatial visualization for investigating the trend and interannual variability typical of ozone profile data.

While this study focused on temporal and vertical variations at a specific location or region, an extension to a spatially varying vertical process (i.e., incorporation of multiple sites with dimensions of longitude and latitude) is theoretically possible. However, four-dimensional correlations could be difficult to disentangle while still maintaining computational tractability. The scaling problem beyond three dimensions could be even more challenging as it involves multiple measurement units in space and time.

The main results of the ozone trend analysis are summarized as follows.

The abundance of IAGOS ozone profiles above Europe is sufficient to provide a reliable trend result without the need for applying an advanced statistical technique. Nevertheless, a sensitivity analysis demonstrates that at low sampling frequencies the integrated fit outperforms the separated fit in terms of accuracy and precision.

A further sensitivity test confirms that regular sampling frequency can be more beneficial than random sampling under the same number of available profiles.

While our method improves trend detection from sparse datasets, the key to substantially reducing the uncertainty is to increase the sampling frequency.

Based on a series of sensitivity studies, we determined optimal sampling frequencies for (1) basic trend detection and (2) accurate quantification of the trend. When applying the integrated fit method, we find that a typical sampling frequency of four profiles per month is adequate for basic trend detection; however, the accurate quantification of the trend requires 14 profiles per month. Accurate trend quantification can be achieved with only 10 profiles per month if a regular sampling frequency is applied. In contrast, the standard separated fit method, which ignores the vertical correlation between pressure surfaces, requires 8 profiles per month for basic trend detection and 18 profiles per month for accurate trend quantification.

Trends derived from the separated and integrated fits are similar above China: stronger positive signals in the upper and lower troposphere and relatively weaker positive signals in the middle troposphere. However, the integrated fit yielded a more consistent trend estimate.

Application of the integrated fit to the Hilo record enables us to effectively reduce the unstructured variation in the profile and derive clear positive trends in the middle and upper troposphere.

The reference year of the trend estimate is particularly crucial at Trinidad Head due to high anomalies in 1998–1999. The trends are weak in the middle and lower troposphere and tend to be more negative in the upper troposphere.

The model proposed in Sect. 2 can be fit to the data, and the coefficients can be estimated by minimizing the penalized least square criterion

On the geometry of a function, the second derivative corresponds to the curvature or concavity of the field; thus this penalty term is designed to prevent a “stand-alone” sharp variability from its neighborhood system. For example, except for the top and bottom layer, the variability in each layer should be bounded by its upper and lower layers, and any variability exceeding these bounds is a potentially short-lived anomaly. Specifically, the penalty term in Eq. (

The solution to the penalized least square in Eq. (

The IAGOS data are publicly available at

The supplement related to this article is available online at:

KLC and ORC contributed to the conception and design. AG, IP and VT contributed to the acquisition of data. KLC conducted the analysis. KLC and ORC drafted the paper, while AG, IP and VT helped with the revision. All authors approved the submitted and revised versions for publication.

The authors declare that they have no conflict of interest.

We thank Toshihiro Kuwayama (toshihiro.kuwayama@arb.ca.gov) and the California Air Resources Board for supporting the ozonesonde program at Trinidad Head in recent years, and Michael Ives at Humboldt State University for his many years of collaboration with NOAA GML and his dedication to launching weekly, and sometimes daily, ozonesondes from Trinidad Head. The IAGOS program acknowledges the European Commission for its support of the MOZAIC project (1994–2003) and the preparatory phase of IAGOS (2005–2013) and IGAS (2013–2016); the partner institutions of the IAGOS Research Infrastructure (FZJ, DLR, MPI and KIT in Germany; CNRS, Météo-France and Université Paul Sabatier in France; and the University of Manchester, UK); the French Atmospheric Data Center AERIS for hosting the database; and the participating airlines (Lufthansa, Air France, China Airlines, Iberia, Cathay Pacific, and Hawaiian Airlines) for transporting the instrumentation free of charge.

This paper was edited by Stefano Galmarini and reviewed by two anonymous referees.