Impact of Nonlinearity on Changing the a Priori of Trace Gas Profile Estimates from the Tropospheric Emission Spectrometer (tes)

Non-linear maximum a posteriori (MAP) estimates of atmospheric profiles from the Tropospheric Emission Spectrometer (TES) contains a priori information that may vary geographically, which is a confounding factor in the analysis and physical interpretation of an ensemble of profiles. One mitigation strategy is to transform profile estimates to a common prior using a linear operation thereby facilitating the interpretation of profile variability. However, this operation is dependent on the assumption of not worse than moderate non-linearity near the solution of the non-linear estimate. The robustness of this assumption is tested by comparing atmospheric retrievals from the Tropospheric Emission Spectrometer processed with a uniform prior with those processed with a variable prior and converted to a uniform prior following the non-linear retrieval. Linearly converting the prior following a non-linear retrieval is shown to have a minor effect on the results as compared to a non-linear retrieval using a uniform prior when compared to the expected total error, with less than 10% of the change in the prior ending up as unbiased fluctuations in the profile estimate results.


Introduction
Optimal estimation is a powerful technique for performing atmospheric retrievals because of its capability to characterize errors and sensitivity (Rodgers, 2000;Bowman et al., 2006).This characterization allows data to be assimilated into chemistry and transport models (Jones et al., 2003) compared to other datasets (Rodgers and Connor, 2003;Worden Correspondence to: S. Kulawik (susan.s.kulawik@jpl.nasa.gov) et al., 2007), and prior vectors to be changed (Rodgers and Connor, 2003).However, these approaches are based on the assumption that the retrieved atmospheric state is spectrally linear with respect to the "actual" atmospheric state, i.e. that a linear expansion of the forward model is accurate between the retrieved and true atmospheric states to significantly better than predicted errors.We test the impact of this linearity assumption on post facto linear operations on TES retrievals such as "swapping" the a priori profile.
Using the most accurate prior will lead to the most accurate results; however conversion to a uniform prior can be useful for scientific analysis, such as highlighting seasonal cycles, comparing observations from two different regions that may have different priors, or comparing results from different satellites.Recent papers which have used TES data linearly converted to a uniform prior include Zhang et al. (2006) who examined the global distribution of TES ozone and carbon monoxide correlations in the middle troposphere, Logan et al. (2008) who studied the effects of the 2006 El Nino on carbon monoxide, ozone, and water, and Luo et al. (2007) who compared TES and the Measurements of Pollution in the Troposphere (MOPITT) instrument carbon monoxide results and explores the influence of the a priori.MOPITT processing currently uses a uniform prior to reduce artefacts arising from the prior and maximize the impact of the satellite data (Deeter et al., 2003).
The Tropospheric Emission Spectrometer (TES), on the Earth Observing System Aura (EOS-Aura) platform, obtains high spectral resolution nadir infrared emission measurements (650 cm −1 -2260 cm −1 , with spectral sampling distance of 0.06 cm −1 for nadir viewing mode) with about 3500 observations every other day (Beer, 2006).The TES data provides profile retrievals for atmospheric temperature, water (Shephard et al., 2008), HDO (J.Worden et al., 2007), Published by Copernicus Publications on behalf of the European Geosciences Union.ozone (H.Worden et al., 2007;Nassar et al., 2008;Osterman et al., 2008;Richards et al., 2008), carbon monoxide (Rinsland, 2006;Luo et al., 2007a, b), and methane, as well as surface temperature, emissivity, and cloud information (Eldering et al. 2008).For details on the TES instrument, see Beer et al., 2006, and for information on the retrieval process see Bowman et al. (2006) and Kulawik et al. (2006a).TES products and documentation are publicly available from the Langley Atmospheric Science Data Center (ASDC), http: //eosweb.larc.nasa.gov/PRODOCS/tes/tabletes.html The TES retrieval strategy is briefly listed to give the reader the context of the TES retrievals of ozone, carbon monoxide, and methane for v003 data.(1) The first retrieval step is a cloud detection step which compares the observed and calculated brightness temperatures in the 11 um region and sets the initial cloud optical depth, and an initial cloud retrieval step is done if the estimated cloud optical depth is large.(2) Atmospheric temperature, water, ozone, surface temperature, cloud optical depths, cloud pressure, and emissivity (if over land) are jointly retrieved.(3) Water and HDO are jointly retrieved.(4) Carbon monoxide, surface temperature, emissivity (if land scene), and cloud optical depths are retrieved.(5) Finally, methane, surface temperature, emissivity (if land scene), and cloud optical depths are retrieved.Note that cloud optical depths and emissivity are re-retrieved for each new spectral region, as they vary by frequency.After each step is completed, the retrieval is assessed as to whether it has converged.Some of the quantities which are calculated and compared to thresholds are the radiance residual and mean, changes in the retrieved surface temperature or emissivity, the amount of signal remaining in the residual; and other known issues (Osterman et al., 2006).A master quality flag, speciesRetrievalQuality, combines the checks into a single quality flag, and is written to the TES product files.Statistical comparison between non-linear retrievals using a uniform prior (SU) vs. conversion to a uniform prior using Eq.(??) (SSC).The black line shows the histogram of the Fractional difference of (SSC -SU) for 3 different pressure levels.The green dashed line is the mean TES reported total error.The lower right plot is the standard deviation of the VMR fractional difference averaged over the entire profile.
A retrieved profile can be expressed as a first order expansion in (x−x a ) (Rodgers, 2000;Bowman et al., 2002): where x a , x, and x are the prior, retrieved, and true profile state in log(volume mixing ratio (VMR)), A is the averaging kernel matrix (Backus and Gilbert, 1970;Rodgers, 2000) which describes the sensitivity of the retrieval to the true state, and ε represents the error resulting from spectral noise, spectroscopic errors, cross-state error, and inaccuracies of non-retrieved species, as discussed in Worden et al. (2004).
Adjustment to a new prior can be done using the following equation (Rodgers and Connor, 2003): where x a and x a are the original and new priors, respectively, x is the original retrieved value, and x is the retrieved value with the new prior.Equation (2) shows that when averaging kernel matrix, A, is unity then changes to the prior have no effect on the retrieved value.Conversely when the averaging kernel matrix is zero, Eq. ( 1) shows that the retrieved state is equal to the prior.The averaging kernel is almost always somewhere in between these two extremes for atmospheric retrievals.In the case of TES, the retrieval vector x includes not only the trace gas of interest, but also surface and cloud properties, and for the ozone retrieval, also water and temperature.When x a is modified for only the trace gas of interest, Eq. ( 2) shows that the propagation to x for the trace gas of interest is the same whether the full retrieval vector is considered or whether the matrices and vectors in Eq. ( 2) refers to just the trace gas of interest.Equation (1) assumes not worse than moderate nonlinearity between the retrieved state and the true state while Eq. ( 2) assumes not worse than moderate non-linearity between the two retrieved states (Rodgers 2000).As a consequence, the averaging kernel derived from a non-linear optimal retrieval with a priori, x a , should be sufficiently close to an averaging kernel derived from a non-linear optimal retrieval with a priori, x a .This linearity assumption is tested with a day's worth of TES data.For non-linear optimal estimates, the initial guess used in the minimization does not af- fect the solution as long as that solution represents the global minimum.On the other hand, if a local minimum is reached, then neither Eq. (1) nor Eq. ( 2) may be valid and the estimated profile will depend on the choice of the initial guess.The dependency of the retrieval on the initial guess is tested as well by also comparing standard retrievals to those that are retrieved using a globally constant initial guess.
Additionally, the reader should be aware that the choice of prior will affect the predicted error in the retrieval through the smoothing error component, which depends on the a priori covariance matrix.The a priori covariance matrix is the expected covariance between the prior and the true state; if the global mean is chosen as the prior, the variance between the prior and the true state will increase as compared to choosing a more accurate prior that depends on latitude and longitude.It is apparent in Figs. 1 and 2 that the errors in the estimated state are much larger for the globally uniform prior than for the original prior, especially in the polar region where sensitivity is less and the prior has changed a great deal.The increased errors will be the same whether the profile was retrieved non-linearly or estimated using Eq.(2). 4. standard processing converted linearly to a uniform prior using Eq. ( 2) (SSC) The data was processed with prototype software which created products equivalent to the publicly available v003 product, with tightened convergence criteria which will be included in v004 processing.For dataset SS, the initial guess  and the prior are the same and vary by latitude and longitude as described below.For dataset SSC, the standard processing (SS) result is converted to a global uniform prior using Eq. ( 2).Datasets SSC and SU should be equivalent; assuming Eq. ( 2) is valid.Similarly, datasets SS and US should be equivalent since, as seen in Eq. ( 1), the initial guess should not impact the final answer, assuming convergence to the global minimum is achieved.For the global uniform prior or initial guess, the global average was created by taking a linear average over all priors or initial guesses for the run.
The initial guess and prior for atmospheric temperature, surface temperature, and water are taken from the Global Model Assimilation Office (GMAO) (Rienecker et al., 2006).For ozone, carbon monoxide, and methane, the prior/initial guess are taken from a climatological MOZART-3 run (Brasseur et al., 1998;Park et al., 2004) which has averages binned by latitude and longitude bands (typically 10-30 degree latitude bands and 60 degree longitude bands).
To compare datasets quantitatively, histograms were made of the fractional differences defined as: Since x represents Log(VMR), a value of 0.10 for the fractional difference indicates a 10% difference.
We also plot differences between (SSC-SU) versus the amount of change in the prior, which shows whether there is a breakdown in the accuracy of the results if changes to the prior are too large, and shows whether changes in the prior introduce biases in the result.Linear regression is used to calculate the slope of differences between (SSC-SU) versus the change in the prior.
Finally, averaging kernels at the result state are compared between the SSC and SU datasets to see if the reported degrees of freedom are consistent when the prior is swapped.This gives an indication of the relative sensitivities of the signal to the trace gas profile amounts, and whether the error analysis is cross-applicable.

Results
A TES global survey consisting of 1152 globally distributed nadir observations from 20-21 September 2004 was run for three different configurations for the prior and initial guess, as described in the methods section.Following the non-linear retrievals, the standard retrieval dataset (SS) was converted to the fixed prior dataset (SSC) using Eq. ( 2).Figures 1 and 2 show the initial and retrieved values at 681 hPa for ozone and carbon monoxide, respectively, for datasets SS, SU, and SSC.The TES nadir observation locations are shown with white +'s and interpolation is done between the TES observation locations.The TES standard prior for both figures (panel a) is taken from a climatological run of the MOZART-3 model binned by 60 degrees longitude, and 10 degrees latitude.For the ozone prior, shown in Fig. 1, panel (a), enhancements are seen in the Northern latitudes (>60 N) and an enhanced band from South America through southern Africa to Australia (the biomass burning region (discussed in Bowman et al., 2008)), and a minimum is seen north of Australia.The standard retrieval shown in Fig. 1b represents these same patterns with a marked enhancement in the biomass burning region.The constant prior cases (panels c and d) agree remarkably well with each other indicating that the linearly converting the prior is valid throughout most of the data.The features in panels c and d can be confidently attributed to the TES data without preconceptions introduced by the prior; however large differences between panels (b) and (c) or (d) indicate a dependence on the prior rather than the data.The absence or presence of particular points passing quality flags can cause minor changes in the three different results.Most of ozone enhancements between 60 S-60 N remain between the standard processing and the converted prior (Fig. 1b and c) indicating that TES retrievals are sensitive at this pressure level over those regions.Poleward of 60 N, patterns seen in the original prior and the standard retrieval are absent, indicating that TES retrievals are insensitive in those regions.
Figure 2 show the same plots as in Fig. 1, for carbon monoxide.The carbon monoxide prior (Fig. 2a) indicates enhancement over South America and southern Africa (in the biomass burning region), north of 40 N, and over India and southeast Asia.The standard retrieval Fig. 2b displays marked enhancement over the prior in eastern South America and western sub-Sahara Africa, and in eastern Asia.The uniform prior results, panels (c) and (d), show good agreement with each other.The East Asia enhancement is present but muted and the pattern and values in the biomass burning region are very similar between panels (b), (c), and (d), however the CO enhancement poleward of 40 N is markedly reduced in (c) and (d) indicating that TES retrievals have less sensitivity in those regions.
Figure 3 shows global maps of the VMR fractional difference (using Eq. 3) for O 3 and CO at 681 hPa for the SSC and SU datasets.The plots show that outliers occur predominately in the tropics, and to a lesser extent, Antarctica.The pattern may suggest two cloud layers, which occur frequently in the tropics (Zipser, 1969), could contribute to the retrieval variation since TES assumes one cloud layer (Kulawik et al., 2006b), however determining correlations between outliers and atmospheric conditions was not explored further in this paper.

Statistical analysis
To quantify differences, statistical analysis was done on the 681 observations which have good quality flags for all three runs (SS (and by extension SSC), SU, and US).The master quality flag is set to screen out about 80% of the bad cases, but will also screen out perhaps 20% of good cases as well (Osterman et al., 2006).
A histogram of the fractional difference between the SSC and SU datasets shows the overall accuracy of changing the prior using Eq. ( 2) vs. using a uniform prior in the non-linear retrieval.From this histogram several relevant quantities can be calculated: (1) the fraction of the observations that are www.atmos-chem-phys.net/8/3081/2008/Atmos.Chem.Phys., 8, 3081-3092, 2008 within 5% of each other, (2) the fractional difference that encompasses 95% of the observations, and (3) the standard deviation of the fractional difference.

Results for ozone
In Fig. 4, a histogram of the VMR fractional difference, using Eq. ( 3), is shown comparing dataset SSC (the standard retrieval converted to a uniform prior using Eq. ( 2) to SU (the non-linear retrieval using a uniform prior) at 681, 178, 38 hPa, and over the entire profile.Figure 4 shows that for ozone, 70-80% of the SSC and SU results are within 5% difference.It is not surprising that histogram for the 177.8 hPa pressure level has the widest spread among the 3 pressure levels chosen because ozone at that pressure level has an order of magnitude variability due to the variations in the tropopause height; a globally constant value for ozone between 100-300 hPa is very challenging to the retrieval.Note that the errors introduced by changing the prior are small when compared to the TES reported total error (green dashed line in Fig. 4).In comparison, the VMR fractional difference of the prior had a 1-sigma value of 0.41, 1.08 (i.e.108%), and 0.16 at 681, 178, and 38 hPa, respectively, indicating significantly more spread in the prior than in the resulting retrieval.The 1-sigma values for the results are shown in Table 1.The histograms in Fig. 4 all show sharp peaks centered near zero but also show more outliers than would be expected from a Gaussian distribution.To determine if the outlying points are a result of a breakdown in the linear transform in Eq. ( 2) that occurs when the a priori change is too large, the difference (SSC-SU) is plotted versus the change in the prior, averaged over the profile, in Fig. 5. Figure 5 shows no obvious difference between small and large prior changes.In Fig. 5, panel (a) shows the rms of (SSC-SU), and panel (b) shows the mean difference, both averaged over the entire profile.For the rms difference, the slope tells whether, on average, larger differences in the prior lead to larger differences in the results.This slope was 0.10.For the mean difference, the slope indicates if changes in the prior are correlated with the error in Eq. ( 2) predictions.If a positive slope is found, it would indicate that sensitivity is significantly increased at the new convergence location compared to the old location when the change in the prior is positive.The slope of the mean difference was found to be −0.02.Together these results mean that the error in the answer will be less than 10% of the prior's change, and will be unbiased with respect to the prior's direction of change.The lack of bias suggests that the differences are not a function of the choice of the uniform prior; further testing with a globally uniform initial guess in the next section strengthens this conclusion.
To check whether the outliers in Fig. 4 are a result of converging to a different local minimum, a run was done with a globally uniform initial guess (dataset US).The initial guess is the starting location for the retrieval, which iterates until convergence is reached.Since the initial guess is not in-cluded in the cost function, which determines the final solution, it should not affect the retrieval assuming the retrieval gets to the global minimum.However, an initial guess far from true can lead the retrieval to a non-global minimum, and systematic errors in the forward model or observed radiance can roughen the error landscape and introduce local minima.A more complete description of TES retrievals is discussed in Bowman et al. (2006).Theoretically, the initial guess does not influence the results (as seen also in Eq. 1) and dataset US should converge to the same answer as the standard retrieval (dataset SS).Differences in these datasets indicate convergence to different local minima, but we do not know whether either has reached a global minimum.The histograms from this run for ozone are shown in Fig. 6.In general, histograms of SS vs. US show a sharper peak and more outliers than the histograms from Fig. 4. For O 3 at 681 hPa, for example, 17% of observations change greater than the TES reported error compared to 2% for results shown in Fig. 4.
Figure 7 has all "initial guess outliers" removed, and compares remaining observations for datasets SSC and SU."Initial guess outliers" are set to be those where the average rms difference over the profile between SS and US were more than 5%, and represent observations that show a tendency to converge to different minima.Results are shown in Fig. 7 for 681 hPa, and correlations shown for the profile standard deviation.In this case, there are significantly fewer outliers (compared to Figs. 4 and 5).The right plot in Fig. 7 shows that the spread in the prior is still about the same, but that the spread in the result is markedly less.This means that the outliers in Figs. 4 and 5 likely result from retrievals converging to different local minima.Table 1 summarizes the results for Figs. 4, 5, and 7 for ozone.
As discussed following Eq.(2), when a retrieval is not sensitive, it will converge to the prior and exchanging the prior will move the retrieval to the new prior, as seen for retrievals poleward of 60 N in Fig. 1.The effects of changing the prior on the most sensitive points is of interest, so statistics were calculated for only those points with a corresponding averaging kernel diagonal value of 0.04 or greater (retaining only the most sensitive half of the data).For 681 hPa, the number of samples dropped from 648 to 290; the bias increased from 0.01 to 0.02, the 1-sigma value increased from 2.0% to 2.7%, the 3-sigma value increased from 15% to 17%, and the fraction within 5% error dropped from 78% to 65%.For 177.8 hPa and 38.3 hPa, the changes are smaller, for example for 38.3 hPa the fraction within 5% error dropped from 78% to 72%.However the result that the error is unbiased and smaller than the reported total error still holds true for the most sensitive points.

Results for carbon monoxide
For TES retrievals, carbon monoxide is retrieved following the retrieval of temperature/water/ozone steps.Consequently, changes to the temperature, surface temperature, or cloud parameters resulting from the uniform ozone prior will propagate into differences in the carbon monoxide step.Swapping only the carbon monoxide, rather than all the species together, may improve on the results shown in this study.Figure 8 shows the histogram of the fractional VMR change for CO at 383 and 681 hPa (note Figs. 8 and 9 do not have initial guess outliers removed).Additionally results are shown for averages over the entire profile.Carbon monoxide shows fewer outliers beyond 10% than found with ozone.
Results for CO are summarized in Table 2.In comparison, the VMR fractional difference of the prior had a 1-sigma value of 0.30 and 0.17 at 681 and 381 hPa, respectively, indicating significantly more spread in the prior change than in the resulting retrieval.

Results for methane
Methane is also retrieved following the temperature/water/ozone steps, and changes to the temperature, surface temperature, or cloud parameters resulting from the uniform ozone prior will propagate into differences in the methane step.The results seen in this study are likely to be worse than the results from swapping only the methane.Figure 9 shows results at 287 hPa and for the whole profile, and shows that changing to a uniform prior results in less than a 1% difference in methane for 95% of the cases.Results for methane are summarized in Table 3.In comparison, the VMR fractional difference of the prior had a 1-sigma value of 0.06 at 287 hPa indicating significantly more spread in the prior change than in the resulting retrieval.

Error analysis differences when changing the prior
When one changes to a different prior following the nonlinear retrieval, the error analysis available is the one calculated at the original retrieval.This section determines whether this error analysis is accurate by looking the change in the averaging kernel between runs SS and SU.Using the total and individual level degrees of freedom for signal from the averaging kernel diagonal, we compare the mean and the fractional differences for the degrees of freedom.Note that the fractional differences for individual levels are calculated for values greater than 0.001 so as to only compare pressure levels which have sensitivity.
prior, when compared to the expected total error.Histograms of differences between these two methods show a sharp peak centered near zero with some outliers, especially for ozone.Further analysis of the characteristics of the outliers, and comparisons to retrievals with a uniform initial guess indicates that the many of the outliers result from convergence to a local minimum rather than breakdown of the linear conversion in Eq. ( 2).For ozone, the 1-sigma difference is less than 4% for each of three pressure levels studied, and the mean change for all levels is 2.7%.For methane, the 1-sigma change is 0.3% at 287 hPa and 0.3% for the profile average, and for carbon monoxide the 1-sigma change is about 2%.The degrees of freedom comparison between shows a 1-sigma difference of less than 3% for all the species, and shows changes of the averaging kernel diagonal are on the order of 20% for individual levels.

Fig. 3 .
Fig. 3. VMR fraction difference for SSC-SU for O 3 (left) and CO (right) at 681 hPa.These plots show that the outliers occur predominately in the tropics.

Table 1 .
Summary of the differences between the linear vs. non-linear application of a uniform prior for ozone.
*The slope is calculated for the mean difference of the profiles.The other average quantities are calculated for the rms difference.

Table 2 .
Summary of the differences between the linear vs. nonlinear application of a uniform prior for carbon monoxide.
*The slope is calculated for the mean difference of the profiles.The other average quantities are calculated for the rms difference.

Table 3 .
Summary of the differences between the linear vs. nonlinear application of a uniform prior for methane.