Interactive comment on “ Global evaluation of the Collection 5 MODIS dark-target aerosol products over land ”

Responses to Reviews We would like to the reviewer for their time in reading and reviewing this lengthy paper. We appreciate their comments and have endeavored to address each recommendation seriously. Only the comments that require revision/rebuttal are included here. REVIEWER #1 . . .Therefore, I recommend the publication of this paper in ACP with minor corrections. My minor concerns are the following ones: (i) authors have to explain/state more explicitly what do they mean by “quantitative”


Introduction
As aerosols are major components in Earth's global climate system, their global distribution must be characterized in order to understand their impacts.The climate and aerosol communities are increasingly relying on satellitederived aerosol data, for research as well as for monitoring.Aerosol products from NASA's Moderate Imaging Spectrometer (MODIS, Salomonson et al., 1989) sensor were utilized in the latest IPCC (4th) assessment of climate (IPCC, 2007), and are being assimilated into chemical transport models (e.g., Zhang et al., 2008).Satellite aerosol products, including those from MODIS, are also being used for estimating and monitoring ground-level particulate matter (PM) at regional and local scales (e.g., Al-Saadi et al., 2005;von Donkelaar et al., 2010).
There are two MODIS sensors (King et al., 2003), observing Earth from polar orbit aboard NASA's Terra (since February 2000) and Aqua satellites (since June 2002).MODIS is uniquely suited for characterization of aerosols, combining broad swath size (∼2330 km), multi-band spectral coverage (36 wavelength bands, spanning from 0.415 µm to 14.5 µm) and moderately high spatial resolution (1 km, 0.5 km, or 0.25 km, depending on band) imaging.Orbit stability and radiometric calibration are both rigorously maintained by the MODIS Characterization Support Team (MCST), to within ±2-3% at typical situations (Xiong et al., 2005(Xiong et al., , 2007)).To take advantage of MODIS's sensitivity to aerosol signals, efficient retrieval algorithms have been developed, maintained, and consistently applied to the entire time series of observations.These algorithms operate by matching observed spectral reflectance (statistics of noncloudy pixels) to lookup tables (LUT) that simulate spectral reflectance for expected aerosol conditions.Each retrieved value represents the aerosol conditions in non-cloudy skies, within some expected error interval.The current suite of MODIS aerosol products is derived separately over three environments: (1) dark-surface (far from sun glint) ocean targets (Remer et al., 2005), (2) dark-surface (vegetation; soils) land targets (Levy et al., 2007b), and (3) bright surface (deserts) land targets (e.g., Hsu et al., 2004).
In this paper, we assess the performance of the aerosol products over dark-land targets (environment 2).Retrieved products include total aerosol optical depth (τ or AOD) at 0.55 µm, spectral AOD at 0.47 and 0.65 µm and the fine model weighting (ETA) at 0.55 µm.Derived products include the fine-model AOD (τ f or fAOD) and Ångstrom exponent (α) defined by using AOD values at 0.47 and 0.65 µm.The union of these AOD and size parameters, plus diagnostic parameters and retrieval Quality Assurance (QA), comprise the set of Level 2 (L2) aerosol products.These L2 products are retrieved at 10 km resolution globally, and are contained in data product files, which we denote as M*D04 (MOD04 for Terra and MYD04 for Aqua).These M*D04 files are processed and archived by the MODIS Adaptive Processing System (MODAPS) at NASA's Goddard Space Flight Center, in Hierarchal Data Format (HDF) with parameters stored as Scientific Data Sets (SDS).The most recent dark-target aerosol data are being processed as Collection 5, or C005 for Terra and Collection 51 or C051 for Aqua.
Prior to Terra launch, Kaufman et al. (1997) estimated the uncertainties for MODIS-retrieved AOD.One of their metrics, designated in this paper as expected error (EE), defined an envelope encompassing the sum of absolute and relative AOD errors.Since then, many studies (e.g., Chu et al., 2002;Remer et al., 2005), have attempted to define EE and validate the MODIS products, most recently for the dataset known as Collection 4 (C004).The C004 MODIS-derived aerosol products were compared to global sunphotometer data, and 68% fell within an EE envelope of ±(0.05 + 20%) on a global scale (Remer et al., 2005).However, this and other studies (e.g., Levy et al., 2005) demonstrated that there were locations and conditions where the C004 errors were systematically larger.These errors were of a magnitude that the C004 products were not accurate enough for use in global model assimilation (e.g., Hyer and Reid, 2009).Levy et al. (2007a, b) characterized some of the limitations of the C004 algorithm, and introduced a "second-generation" dark target algorithm to process C005.Although there have been studies using C005 dark-target products both globally (e.g., Remer et al., 2008), and regionally (e.g., Mi et al., 2007;Jethva et al., 2007;Papadimas et al., 2009), this paper is a more in-depth evaluation.Here, we compare the entire MODIS time series from both Terra and Aqua to global AERONET data, thus quantifying global EE, and identifying where and under what conditions the C005 products may still be falling short.In Sect.2, we briefly summarize the C005 dark-target aerosol retrieval and products, and define the concept of EE.We compare the MODIS-derived aerosol products with measurements by ground-based sunphotometers, for spectral AOD in Sect.3, and for aerosol size parameters (including Ångström exponent and fine AOD) in Sect. 4. We use the spatial-temporal collocation method that was introduced by Ichoku et al. (2002), and used previously by Remer et al. (2005) and others.In Sect.5, we summarize our validation results and suggest steps necessary to reduce the remaining systematic discrepancies.Section 6 offers some discussion of the significance of the results and conclusions.

The MODIS aerosol retrieval over land
The MODIS "dark-target" aerosol retrieval algorithm is designed to infer clear-sky (non-cloudy) aerosol properties from MODIS observations, over land surfaces that have low values of surface reflectance (e.g., "dark") in parts of the visible (VIS) and shortwave infrared (SWIR) spectrum.Generally, vegetated and dark soil regions are examples of such dark targets, providing a contrasting surface to observe relatively brighter aerosol reflectance.The basic algorithm uses two visible (VIS, 0.47 and 0.65 µm) and one shortwave IR (SWIR; 2.1 µm) bands (Kaufman et al., 1997;Levy et al., 2007b), which (a) are nearly transparent to CO 2 , H 2 O and other gaseous absorption, and (b) demonstrate a consistent spectral relationship over vegetated land surfaces (Kaufman et al., 1997).Additional wavelengths in other parts of the spectrum are used to mask out clouds, deserts, snow, and ice surfaces -non dark-target conditions (Ackerman et al., 1998;Martins et al., 2002;Li et al., 2003).Note that a vegetated surface is not "dark" in the green MODIS wavelength (e.g., 0.55 µm), and therefore, the 0.55 µm channel cannot be used directly.
The object of the retrieval is to determine the effective columnar aerosol properties, which, coupled with a constrained surface contribution, best represents the MODIS spectral reflectance observations in the three channels.By "best represents", we mean a solution that provides the smallest fitting error when matching the LUT reflectance to the observations.The aerosol is assumed to a mixture of fine and coarse model types, and the primary retrieved products are the total aerosol loading, represented by the AOD defined at 0.55 µm, and the fine model weighting, ETA, also defined at 0.55 µm.Both AOD and ETA represent the integration of aerosol properties through the entire atmospheric column, with no assumptions as to aerosol layering.
The term fine model refers to the physical and optical properties (size distribution, complex refractive indices, shape) of column-integrated aerosol that is dominated by fine mode sized (e.g., radius <<1.0 µm) particles.This is in contrast to a coarse model that is dominated by particles larger than one micrometer.Previously, Levy et al. (2007a) performed cluster analysis of AERONET almucantur inversions (e.g.Dubovik and King, 2000) and identified three fine models and one coarse model that generally represented global aerosol types over dark-land surfaces.The three fine models were spherical and differed mainly in their single scattering albedo (SSA), such that the weakly, moderately and strongly absorbing models had approximate SSA at 0.55 µm of 0.95, 0.91 and 0.86, respectively.The non-spherical, coarse model had SSA = 0.95 and essentially represented dust aerosol type.In addition to identifying three global fine models, Levy et al. (2007a) also determined which fine model was most appropriate for a given season and location, which is assumed by the algorithm.The MODIS algorithm mixes the coarse (dust) model with the appropriate fine model type, thus determining AOD and ETA.
The LUT represents TOA reflectance for aerosol plus molecular (Rayleigh) scattering over a black surface, specific for each aerosol model type.It was previously computed using Radiative Transfer (RT) that includes polarization effects (Levy et al., 2004;Evans and Stephens, 1991).The Modified T-matrix code (Dubovik et al., 2006) was used to calculate the dust-type scattering using spheroids, whereas a Mie-scattering code (Wiscombe, 1980) was used for the spherical fine model particles.The LUT simulations are indexed by seven values of total aerosol loading, indexed by AOD at 0.55 µm (τ = 0.0, 0.25, 0.5, 1.0, 2.0, 3.0 and 5.0).However, the observed TOA signal is made up of reflectance scattered from the aerosol as well as the surface.Kaufman et al. (1997) showed that if the surface reflectance is "known", then the aerosol properties can be retrieved from observations at the TOA.Since observations and modeling studies had shown that surface reflectance was spectrally linked for vegetated and dark-soil land surfaces, then the MODIS aerosol retrieval was possible.Prior to C005, the dark-target algorithm was essentially "single channel retrieval" in two visible wavelengths (0.47 and 0.65 µm).The 2.1 µm aerosol contribution was assumed to be negligible (e.g., Kaufman et al., 1997), so that the surface reflectance at 2.1 µm was "known".Using simple ratios of this 2.1 µm surface reflectance, the surface contribution to the two visible channels could be estimated, and its contribution to the TOA reflectance accounted for.The measured reflectance at top-of-atmosphere (TOA) at 0.47 µm and 0.65 µm was separately compared to simulated TOA reflectance in these two channels, so that total AOD was determined independently at each wavelength.Calculation of the final product, the AOD at 0.55 µm, was accomplished by interpolating between retrieved AOD at each channel.
Today, the algorithm is more of an "inversion".The atmospheric LUT represents spectrally consistent atmospheric properties computed for each aerosol model coupled with molecular (Rayleigh) scattering.Thus, for a given aerosol model, the spectral dependence of any aerosol property is fixed.Since dust aerosol reflects at 2.1 µm, the algorithm does not need to assume that the aerosol contribution to the observed 2.1 µm is negligible.At the same time, the spectral surface reflectance contribution is constrained by the surface reflectance parameterization described by Levy et al. (2007b), and is coupled to the atmosphere LUT.Since the LUT is spectrally consistent, when the algorithm matches the measured reflectance at 0.47 and 0.65 µm to calculated values from the LUT, the AOD at any wavelength is automatically determined.Thus, although the MODIS-observed 0.55 µm reflectance is not used directly within the inversion, the AOD value at 0.55 µm can be retrieved directly and without interpolation.Levy et al. (2007b) chose to report 0.55 µm as the primary wavelength because it falls between the two channels that contribute information.Also, 0.55 µm is consistent with the primary wavelength used by models and other community-wide applications, including the MODIS retrieval over ocean (e.g., Remer et al., 2005).
For elevated targets, the phase function dependence of Rayleigh/aerosol interaction is also accounted for, rather than simple subtraction of Rayleigh optical depth as was applied for C004 and prior (Levy et al., 2007b).Thus, the LUT remains spectrally consistent.
In addition to the assumptions related to inversion, aerosol model properties and surface characteristics, the C005 algorithm is fundamentally different in how it reports aerosol properties in very clean conditions.To reduce statistical biases in low-AOD conditions (AOD < 0.05), negative AOD values down to −0.05 are permitted.
Although the nominal resolution of MODIS is 500 m in most wavelength bands, the MODIS aerosol retrieval is performed at 10 km.The 10 km retrieval allows us to improve the signal-to-noise ratio, since we can throw out many pixels (e.g., containing clouds, cloud shadows, snow, or surface inhomogeneities) and still have sufficient clear-sky statistics for doing the retrieval (Levy et al., 2009).Other than implementing the snow mask of Li et al. (2003) into the C005 algorithm, the pixel selection technique remains the same as for C004.The 3 × 3 visible reflectance variability test (e.g., Martins et al., 2002) provides the primary cloud screening, and of the remaining pixels, 20% of the darkest and 50% of the brightest pixels are discarded.
In addition to aerosol properties, the MODIS algorithm reports a number of diagnostic products, including a protocol for assessing the "Quality" of the retrieval known as the Quality Assurance (QA) plan (e.g., Hubanks, 2007).The QA plan is a series of tests that indicates whether certain conditions are met during the course of the retrieval.At the end of the retrieval process, a summary QA Confidence (QAC) flag recaps the results of all QA tests, and indicates a relative "Confidence" in the entirety of the retrieved product.QAC ranges from 3 (high confidence) to 0 (low or no confidence).For C005, there were major changes to the QA plan (Levy et al., 2009).The new QA plan includes information characterizing the type, quality and confidence of the input MODIS reflectance data, ancillary datasets (e.g., meteorology or ozone ancillary data), as well as some of the intermediate and output products (Levy et al., 2009).We expect that data having larger QAC values will be more comparable with ground truth, and, therefore, be more useful for scientific applications (e.g., Kahn et al., 2009).

Clarification of aerosol size parameters
Together, AOD and ETA parameters (both at 0.55 µm) are the retrieved aerosol solution to the algorithm, representing the total column aerosol loading and the effective proportion of fine and coarse aerosol model types in the column.Because the LUT is spectrally consistent, AOD is also determined at 0.47 and 0.65 µm.Based on these two channels, the Ångstrom exponent (AE or α) can be derived.The AE is a one-parameter description of the spectral AOD dependence, which can be related to relative aerosol size (e.g., Eck et al., 1999).Larger values of AE (steeper spectral dependence) indicate smaller column-effective particle size, and conversely.
The ETA parameter is difficult to interpret.Unlike the finemode weighting parameter retrieved by the MODIS overocean algorithm (e.g., Tanré et al., 1997), ETA represents a relative mixing of fine and coarse models within the column.Since both land aerosol types (fine-dominated and dust) have multiple modes, and the retrieval solution is strongly influenced by the assumptions of spectral surface reflectance, ETA over land does not represent a clear division of particle sizes, nor is it entirely independent of surface features.Yet, based on sensitivity tests and comparisons with AERONET (e.g., Levy et al., 2007b), retrievals of AOD are more accurate when the retrieval has the flexibility to combine two models weighted with the ETA parameter than when forced a priori into a single aerosol model.Therefore, although one might expect some correlation between ETA and AERONET-derived fine mode weighting, we do not expect this to be true in a global sense (e.g., Levy et al., 2007b).ETA cannot easily be compared to quantities reported by AERONET (e.g., O'Neill et al., 2003).
Even though Levy et al. (2007b) concluded that ETA would be a difficult quantity to retrieve and validate, they suggested that particular derived products might be more useful.For example, they suggested that the product of AOD and ETA, known as the fine-model AOD (fAOD or τ f ) could be used for comparison with models or estimating aerosol radiative forcing.The fAOD is reported only at 0.55 µm.Note that the fAOD quantity is the fine-model AOD because one of the multi-modal models is dominated by the fine mode.(The other, the dust model, is dominated by the coarse mode).Since Levy et al. (2007b) found that MODISderived fAOD was correlated with AERONET-derived finemode AOD, they suggested it should be reported within the MODIS product.However, as we report in Sect.3.3, we now conclude that although ETA has important diagnostic value, in general the MODIS size parameters are too burdened by a priori assumptions and surface reflectance uncertainty to be quantitative on the global scale.

Measures of evaluation
Because the MODIS algorithms are designed to infer aerosol properties from the radiation field, uncertainties in the retrieval assumptions and retrieval methodology lead to uncertainties in the retrieved products.The question is whether the MODIS products are quantitative.By quantitative, we refer to a product that (a) we can validate, (b) has statistically significant correlation with an observable physical quantity and (c) we believe can be used in a scientific study or application relating to understanding of a geophysical quantity or process.By validation, we mean that we can assign wellcharacterized error bars to the product such that 2/3 of the values, globally, match with ground truth (e.g., AERONET) -observed properties.
Prior to Terra launch, Kaufman et al. (1997) used sensitivity studies to estimate the expected error (EE) of the MODISretrieved AOD.Estimated as ±(0.05 + 0.20τ ), the MODIS EE represented the fusion of absolute (0.05) and relative (20%) uncertainties that would arise from combined errors in assumed boundary conditions (e.g., surface reflectance, instrument calibration) and errors in aerosol model type (such as in single scattering albedo).However, some outliers are to be expected, and a key objective of the post-launch validation process has been to determine the size of the envelope needed to contain 66% (one standard deviation) of actual retrievals globally, and under different conditions.
Since the launch of Terra (and later Aqua), the actual MODIS-derived AOD was repeatedly collocated with, and compared to global sunphotometer data, which is used as ground-truth (e.g., Chu et al., 2002;Remer et al., 2005).Good matches were reported wherever the MODIS -retrieved AOD, τ MODIS , fell within the envelope defined by Through global comparison with AERONET, Chu et al. (2002) suggested that the EE could be reduced to which was later confirmed by Remer et al. (2005) for a large dataset (Collection 4; 5906 collocations).In this current paper, we will determine the degree to which the EE envelope represented by Eq. ( 2) also represents C005.
As for MODIS-reported size parameters, they have never been quantified over land.Global EE has not been determined for any size parameter, because of the expectation they would not be sufficiently accurate to warrant such effort (e.g., Remer et al., 2005).Given the difficulty in retrieving and interpreting ETA, we do not expect to validate it for C005, either.However, we perform the exercise of deriving EE for these products and report the results below.
Equation (2) represents the EE for global AOD.In addition to Remer et al. (2005) other studies (e.g., Levy et al., 2005;Hyer and Reid, 2009) (Holben et al., 1998) sunphotometer measurements (>1200 cases) was improved, as demonstrated by the correlation coefficient (R) increasing from 0.85 to 0.89, and the y-offset decreasing from 0.097 to 0.029.For the testbed, 67% of the MODIS/AERONET AOD collocations fell within the EE envelope (Eqs. 1 and 2), indicating preliminarily validation.Levy et al. (2007b) also performed some preliminary evaluation of the C005 algorithm's reported size products, by comparing with values reported by AERONET.The ETA (fine-model weighting) parameter was shown to have essentially no correlation with AERONET-derived fine-mode weighting.There was also no significant correlation for AE.Yet fine AOD was correlated (R = 0.88) and generally agreed to within EE defined by Eq. ( 2).
Although in general, the C005 algorithm products compared to AERONET at least as well as those derived by the C004 algorithm, we note that the C005 algorithm used archived C004 radiances.During operational processing of the C005 algorithm, MCST introduced new calibration coefficients (http://mcst.gsfc.nasa.gov/uploads/files/c5 luts update/L1B Aqua LUT History.txt) and (http://mcst.gsfc.nasa.gov/uploads/files/c5luts update/ L1B Terra LUT History.txt).This means that the preliminary validation performed by Levy et al. (2007a) may not apply to actual C005 products.In fact, Remer et al. (2008) identified differences between the C004 and C005 datasets over ocean that could be attributed to the calibration changes.Thus, we perform a full evaluation of the actual C005 dataset.

Global evaluation of C005 products
The algorithm's retrieved parameters are solutions to the lookup table matching.
These are AOD and ETA (at 0.55 µm), and are reported as SDSs in the M*D04 file as "Corrected Optical Depth Land" and "Optical Depth Ratio Small Land", respectively.The fAOD is simply the product of the solution at 0.55 µm (AOD × ETA) and is reported as "Optical Depth Small Land".Calculation of additional parameters requires the information embedded in the LUT.For example, based on assignment of aerosol model type, which is in turn associated with assumed aerosol optical properties (e.g., spectral extinction), we back out the spectral AOD (0.47 and 0.65 µm; "Corrected Optical Depth Land" SDS) and compute the AE ("Angstrom Exponent Land").
The Quality Assurance (QA) information, including the summary QAC, are diagnostic information, reported using the "Quality Assurance Land" SDS.Tables of the QA tests are found in Levy et al. (2009b) and Hubanks (2007).Kahn et al. (2009) also describe the MODIS QA flags; note that QAC (this study) and their QC flag are the same quantity.The QA Usefulness (QAU) flag (1st bit) is necessary for Level 3 processing (e.g., Hubanks et al., 2007).Depending on which tests pass or fail, the algorithm will report "fill value" (i.e., no data) for all, some, or none of the parameters.For example, if the scene reflectance is brighter than 0.25 at 2.1 µm, only the spectral AOD is reported.All the other parameters are set to fill values and QAC is set to zero.Independent of the QAC, if the retrieved AOD is small (e.g., τ < 0.2), the aerosol signal is probably too low to retrieve meaningful aerosol size information, Thus, fAOD, ETA, and AE would be reported as missing.

Collocation with AERONET
Here, we collocate the entire set of C005 Terra/Aqua-MODIS aerosol retrievals with the AERONET Version 2.0, Level 2 Quality Assured (cloud screened and calibrated) direct-sun measurements of spectral AOD (Holben et al., 1998;Smirnov et al., 2000).The AERONET AOD uncertainties are on the order of 0.01-0.02(Eck et al., 1999), and we consider them as "ground truth" for satellite product validation.Using quadratic fits on a log-log scale (Eck et al., 1999), we interpolate the AERONET data to MODIS band-effective wavelengths (i.e., 0.47, 0.55 and 0.65 µm bands), and calculate the 0.47/0.65 µm Ångstrom exponent to match that reported in the MODIS product.Finally, we use the spectral de-convolution technique of O' Neill et al. (2003) to derive AERONET fine mode fraction and finemode AOD, and use these to show that globally, the MODIS size-parameters have little global quantitative value.
We employ the spatio-temporal technique of Ichoku et al. (2002), which creates a grid of 5 by 5 MODIS aerosol retrieval pixels, with the AERONET station in the middle.
Since each MODIS aerosol pixel represents approximately a 10 km area, the subsetted 5 × 5 area is approximately 50 km by 50 km.Spatial statistics for the MODIS subset are calculated and compared to the temporal statistics of the AERONET observations taken within ±30 min of MODIS overpass.At least 5 of the possible 25 MODIS retrievals, and 2 of the possible 4 or 5 AERONET observations, are required to include a collocation in our statistics.This means that the collocation might not include the exact 10 km MODIS aerosol retrieval region in which the AERONET station resides, and could include retrievals from pixels that are 20-25 km away.For a collocation to be included, both MODIS and AERONET require sampling that is sufficiently free of clouds, based on their respective cloud-masking algorithms.This precludes evaluation of MODIS products in conditions of overcast or some partial cloud situations.
As of September 2008, our database included collocations with 328 AERONET sites, of which 32 were island sites that could not be used for over-land comparison.Of the remainder, 203 sites were inland, and the rest located at or near shoreline.Some sites offer long measurement time series, whereas others have measurements only during particular seasons or field experiments.To exclude sites where the AERONET instrument is not likely to represent the aerosol of the surrounding region, we compared the altitude of the AERONET site and the average digital elevation (http://www-surf.larc.nasa.gov/surf/pages/digelev.html) of the surrounding 50 km × 50 km box.To retain sufficient global sampling, we excluded only the sites where the AERONET/box elevations differed by >300 m.The result is 85 463 matches for the combined Terra/Aqua dataset.For discussion in this paper, we will label sites by the names given by the AERONET team (http://aeronet.gsfc.nasa.gov).

Global AOD
Figure 1 is a frequency scatterplot of the over-land comparison of total AOD at 0.55 µm, for the union of Terra and Aqua datasets ("Both").The data are not filtered by QAC.The color of each ordered pair (0.025 × 0.025 increment) represents the number of such matchups.Each axis ranges from −0.1 to 3.0.The dashed, dotted and solid lines are the 1-1 line, defined EE for land AOD (Eq.2), and the linear regression of the pre-sorted scatterplot, respectively.Table 1 gives some of the regression statistics.Sixty-nine percent of the MODIS retrievals of the comprehensive data set of 85 463 collocations lie within the EE defined by Eq. ( 2).In addition, as compared to C004 (Remer et al., 2005), the C005 AOD (at 0.55 µm) shows much closer match to the 1-1 line (y = 0.95x + 0.005 vs. y = 0.78x + 0.068), and a higher correlation (R = 0.88 vs. R = 0.80).Validation is also achieved for the 0.65 and 0.47 channels (also in Table 1).

Global aerosol particle size
Figure 2 shows the scatterplots of MODIS-derived AE (top) and fine-model AOD (bottom) against the respective AERONET quantities of AE and fine-mode AOD.Containing one standard deviation (66%) of the collocated points, the expected errors (EE) for AE and fAOD are plotted as ±0.4 and ±(0.05 + 0.20τ f ), respectively.
Figure 2 (top) is similar to Fig. 3a of Liu and Mishchenko (2008), but by showing the density of points, we illustrate that in fact the MODIS product is generally binary; either it derives AE close to 1.8 or AE of exactly 0.5.Values of AE close to 1.8 indicate that retrieved ETA is exactly 1.0 (fine-model only), whereas AE of 0.5 indicates that retrieved ETA is exactly 0.0 (coarse/dust model only).There are very few cases where MODIS-reported AE is in between, indicating that the MODIS retrieval cannot capture the variability of the ground-truth AE.In most cases, when MODIS indicates the dominance of the fine aerosol model, it agrees with AERONET.However, the converse is not true; MODIS finds dust where there should be none (e.g., Jethva et al., 2008;Mielonen et al., 2010).Although we can define global EE for AE as ±0.4, the combination of binary behavior and unrealistic values clearly indicates that AE is not globally quantitative.
Figure 2 (bottom) shows that there is apparent correlation (R = 0.81) of the fine-model AOD with the AERONETderived fine-mode AOD.However, we realize that because Atmos.Chem.Phys., 10, 10399-10420, 2010 www.atmos-chem-phys.net/10/10399/2010/retrieved ETA is effectively binary, the apparent validation of the fAOD retrieval is entirely due to the strength of the total AOD retrieval (e.g., Fig. 1).Therefore, fAOD cannot be globally quantitative either.Based on the above analysis, we conclude that the C005 aerosol size parameters have little physical validity, whether obtained directly (ETA) from matched radiances, or derived downstream (AE or fAOD).In future collections of MODIS data, we recommend that fAOD and AE be removed to avoid confusion.ETA should be retained, because it provides important information about algorithm performance.Users can derive fAOD and AE for a particular application, with the understanding that they primarily reflect algorithm assumptions about particle type.In this paper, subsequent reference to AE refers to AERONET-measured values.

Dependence on algorithm QAC
As mentioned previously, each set of retrieved products is accompanied by the algorithm's estimate of the QAC, which may range from 3 (high confidence) to 0 (low or no confidence).Table 2 presents the information contained in the 0.55 µm row of Table 1, but stratified by QAC value.We see that the quality of the MODIS/AERONET comparison is strongly dependent on QAC.For the retrievals with QAC = 0, there is significant deviation from the 1-1 line.Although the global averages of both datasets are similar for QAC = 0, MODIS retrieves only 50.34% of the cases to within EE of ±(0.05 + 15%).However, as we increase our QAC value, the regression become more symmetric to the 1-1 line and the percentage within EE increases to 66.10%, 67.75%, and 72.60% for QAC = 1, 2 and 3, respectively.In general, retrievals with QAC = 3 provide the best matches to AERONET, so all further analyses in this paper will be performed for those 58 526 points.Note that a user may choose to use data with lower QAC, but is strongly advised not to use retrievals with QAC = 0 for any quantitative purpose.The choice depends on the application's tolerance for uncertainty versus the need for spatial coverage.
Figure 3 presents the 58 586 QAC = 3 cases of C005, binned by AERONET AOD.On the x-axis, there are 50 bins, each containing ∼1200 observations.The y-axis is the absolute difference between MODIS and AERONET AOD (MODIS-AERONET.The statistics are presented as boxwhisker plots, where the horizontal centers and half-widths of the red boxes represent the means and the standard deviations of the AERONET AOD in each bin.In the vertical, the centers and the tops/bottoms represent the medians and the middle 66% (1σ ) intervals of the MODIS-AERONET differences for each bin.The black squares are the mean of the MODIS-AERONET differences (usually close to the median).The red dashed-dot lines are linear best fits to the bottoms and tops of the boxes, which can be compared with the green dashed lines encompassing the EE envelope.Finally, the blue whiskers represent the 96% (2σ ) intervals of the MODIS-AERONET differences.
From Fig. 3, we see across the entire AOD range that: (a) the mean bias of the MODIS retrieval is generally less than 0.01, and that (b) the 66% interval and the green EE envelope are nearly identical.This means that for cases with QAC = 3, the EE is a reasonable assessment of C005's AOD error across the entire range of AOD.We note that the 2σ bars are approximately double of the 1σ (and EE) envelope.

Local C005 validation
Although we have robustly validated the global C005 AOD within EE, we know from the literature (e.g., Jethva et al., 2008;Kahn et al., 2009) that there are regions where MODIS has systematic problems retrieving AOD.MODIS may systematically overestimate or underestimate AOD for one reason or another, and the errors might be offsetting (Kahn et al., 2007).In this section, we examine the performance of the MODIS algorithm for retrieving AOD at individual AERONET sites.By separating into cases with light aerosol loadings (τ < 0.15) and heavy aerosol loadings (τ > 0.4), we can distinguish between cases where the systematic errors result from poor surface assumptions and cases where sys- tematic errors result from poor aerosol model assumptions.Although the separation can never be complete, surface assumptions tend to dominate the retrieval when there is little aerosol, whereas the surface becomes less important as the aerosol load increases.

Site by site: overall
Our dataset includes collocations from different sites and different seasons.These sites represent a variety of surface types (forests, savanna, urban, soils, etc.), and a variety of expected aerosol types.Although we have demonstrated that on average, the retrieval algorithm has made the correct assumptions as to surface and aerosol characteristics, we know that there are sites where MODIS shows systematic errors.
Let us consider the fraction of the MODIS-retrieved AOD values that fall within EE at each site during a given season, as well as the sign of the mean bias.Where we see at least 2/3 (66% or 1σ ), we consider this to have "good" matching.If fewer than half lie within EE, this is a "poor" match, and we also note whether MODIS tends to retrieve too low or too high.Figure 4 provides visual assessment of both matching quality and MODIS bias, during the summer months (June, July, August).Although examined for other seasons, only the summer months are shown here because there are more aerosol hotsptots (Northern Hemisphere pollution, Saharan dust, etc.) during this season.Symbols are plotted at AERONET sites having at least ten collocations for the season, and are color-coded based on the fraction of MODIS data that matches within EE. Green symbols are plotted where ≥66% match ("good") within EE (e.g., GSFC).Red represents sites where <50% match within EE and MODIS is biased high (e.g., Dalanzadgad).There are no cases for this season, but if plotted, purple would refer to sites for which <50% match and MODIS is biased low.Alta Floresta is marked by yellow because the fraction is between 50% and 66%, with a high bias.Jabiru is marked blue to represent a similar fraction, but biased low.Scatterplots for four example sites GSFC (38 • N, 76 • W), Alta Floresta (9 • S, 56 • W), Dalanzadgad (43 • N, 104 • E), and Jabiru (12 • S, 132 • E) are displayed at the bottom, indicating why the site received a certain color symbol.
For the summer months, much of the US East Coast shows very good agreement at 0.55 µm.An exception is New York City (the CCNY and GISS sites, both near, 40 • N, 73 • W), where the urban surface is known to be poorly represented by MODIS's surface reflectance parameterization (e.g., Oo et al., 2010).Similar urban surface problems are also encountered elsewhere (e.g., Jethva et al., 2007).Most sites in Western Europe also compare well, except for the Venise site (45 • N, 12 • E), which is plotted as a land site, but the AERONET instrument is actually on an oceanographic platform in the Adriatic Sea, some 15 km from an urban area.Essentially, since the MODIS C005 algorithm was developed based on MODIS/AERONET collocations and AERONET sky retrievals available through 2005 (Levy et el., 2007a, b), the US East Coast and Western Europe dominated the database.Except for the urban and offshore sites, the surface is generally at least partially vegetated, and the regional aerosol is characterized by fine-dominated particles with a high single scattering albedo.Thus, it is not surprising that the products generally compare well in these regions.
Good comparisons are seen over southern Africa (e.g., Mongu, 15 • S, 23 • E) and parts of the Amazon for this season, which are both dark surface regions and were well sampled by AERONET prior to C005.Even though Japan and Korea were not well sampled prior to C005 development, there is generally good agreement.This is most likely because the surface is not too different from that present over the well-characterized Eastern US and European regions, and the AOD is not large enough so that a wrong choice in aerosol model would be catastrophic.Good agreement is also seen at the Chinese sites of Taihu (31 • N, 120 • E) and Xianghe (39 • N, 116 • E) (Mi et al., 2007).Interestingly, while the region surrounding Kanpur, India (26 • N, 80 • E) is relatively bright, the sunphotometer site is located in a small (1 km × 1 km) pocket of vegetation.Although the vegetated area is probably too small for the entire scene to be considered "vegetated", there is generally good agreement in the summer (e.g., Jethva et al., 2007).
In addition to the urban surfaces mentioned above, MODIS compares poorly over brighter and elevated targets.For example, Palencia, Spain, is on a plateau, and has a relatively brighter surface where the aerosol signal is comparatively weak for a dark-target retrieval.Poor comparisons are also noted at sites over the western US, (e.g., BSRN-Boulder, 40 • N, 105 • W) and Sevilleta (34 • N, 106 • W), the Patagonian region of Argentina (e.g., Trelew, 43 • S, 65 • W), and the steppe and near desert plateaus of Russia and China (e.g., Irkutsk, 51 • N, 103 • E and Dalanzadgad).While these scenes do not exceed the brightness criteria test for dark target scenes (2.1 µm reflectance greater than 0.25), our surface reflectance parameterization may be violated.In addition, these regions may be dominated by aerosol types that would not have been characterized by the clustering of AERONET data available in 2005 and assumed for the dark-target algorithm.These regions may be better suited for retrieval with the Deep Blue algorithm (Hsu et al., 2004), but testing this hypothesis is beyond the scope of this paper.
Although most of the darker-target sites compare to within global EE as expected, we find sites that compare less well.For examples, Alta Floresta and Cuiaba (15 • S, 56 • W) are both in Brazil, one near the border of the Amazon forest, the other located further south in the cerrado (savannalike vegetation).These two sites have been collecting data since MODIS launch and were used when developing C005.Yet, for both sites, AOD tends to be overestimated in heavy aerosol conditions, and underestimated (consistently negative) in light loading conditions.These differing biases result from poor assumptions for both the aerosol model and the surface reflectance.During the development of the C005 aerosol models, Levy et al. (2007b) found that the aerosol type in the region had varying SSA, jumping between the moderately and strongly absorbing aerosol models.Because there was a tendency for lower SSA towards the southeast, www.atmos-chem-phys.net/10/10399/2010/Atmos.Chem.Phys., 10, 10399-10420, 2010 they drew an a box to signify where the strongly absorbing type should be preferred.However, because the box borders were somewhat arbitrary, we believe that the box was drawn too far west, leading to systematic overestimate in heavy aerosol conditions, especially at Cuiaba.Analysis of Version 2 AERONET retrievals (http://aeronet.gsfc.nasa.gov)suggested that, in fact, SSA over Cuiaba is normally closer to 0.9 than to 0.86 during the dry season.At Alta Floresta (farther west), where the C005 moderately absorbing aerosol type (SSA ∼ 0.91) is assigned, the true SSA is closer to 0.92-0.93(Schafer et al., 2008).Therefore, these SSA discrepancies may at least partially explain why both sites would experience similar systematic bias.A correction to the aerosol model assignments in Brazil is required, in some ways opposite to the correction that was implemented over Southern Africa (Ichoku et al., 2003) for C004.As for the consistent retrieval of negative AOD in light loading conditions, these two Brazilian sites may suffer from similar problems as noted at Jabiru (Northern Australia).This systematic bias for low AOD results from overestimating the surface reflectance in the visible channels.Since the C005 algorithm was optimized for the set of global collocations that favored sites in the Eastern US and Western Europe, the surface reflectance parameterization was biased to-ward these sites and their NDVI characteristics.The vegetation in the Amazon rainforest has smaller visible/SWIR ratios than the presumed global average.In addition, parts of the Amazon (as well as Australia) are known to have red soils, which may not display the same surface reflectance relationships as modeled with the C005 parameterization.

Separating surface assumption and aerosol assumption errors
At Cuiaba and Alta Floresta, the MODIS-derived AOD are overall within EE, but that general assessment hides offsetting biases related to surface and aerosol assumptions.To evaluate these issues we separate the MODIS/AERONET comparisons into three groups, based on the AOD (at 0.55 µm) observed by AERONET.Collocations where τ < 0.15 are "light" aerosol loading conditions, for which the surface contribution dominates the TOA signal.For these cases, we can diagnose MODIS retrieval errors dominated by surface reflectance assumptions.On the other hand, cases where τ > 0.4 are "heavy" aerosol loadings, for which the surface signal is negligible compared to the aerosol signal.For these cases, we can focus on MODIS retrieval errors related to aerosol model assumptions.The cases of 0.15 ≤ τ ≤ 0.4 likely are influenced by both surface and aerosol errors, so we will ignore these cases for this exercise.Of course, there are also regions, such as the borders of the Sahara, where there is heavy aerosol, and we also expect there to be problems with our dark surface assumptions.
Here we can compute an error ratio (ER), which compares the actual error to the expected error (e.g., Mi et al., 2007).Where −1 ≤ ER ≤ 1, the actual error is smaller than EE, indicating a "good" match.Where |ER| > 1, it is a "poor" match.MODIS underestimation and overestimation are represented by ER < 0 and ER > 0, respectively.For each group (light and heavy loadings) separately, we calculate the mean ER of the MODIS/AERONET matches (minimum of ten) at each site and season, and use these values to characterize the relative quality of the MODIS product.Figure 5 is a color-coded map of the mean ER at each site during summer months, for the light (τ < 0.15; top panel) and heavy (τ > 0.4; bottom panel) aerosol cases.The cyan and green colors are sites where |ER| ≤ 1, meaning that the systematic bias is less than the EE for the particular AOD group.Cooler colors indicate ER < 0 (MODIS underestimation) whereas warmer colors represent ER > 0 (overestimation), and at the extremes, purple and red represent increasingly severe average biases.Unlike single collocation estimates of ER, sites where average |ER| > 0 indicate systematic bias to the MODIS retrieval.
Separation by aerosol regime helps to provide confirmation of our hypotheses in the previous section.For example, in the Amazon, MODIS clearly underestimates AOD in light loading conditions, and overestimates in more polluted conditions, indicating both that the surface is darker (in the visible) than the VIS/SWIR relationship suggests, and that the particles are brighter than that assumed for the region.In urban or coastal areas (e.g., CCNY or Venise) and brighter elevated surfaces (e.g., US Southwest) the MODIS overestimations are generally confined to the low AOD conditions, indicating that the surface assumptions are the dominant sources of error.For the polluted conditions during the summer months, MODIS underestimates AOD in the biomass burning regions of the African Sahel, Dakar (14 • N, 16 • W) and Ouagadougou (12 • N, 1 • W), which we believe is primarily a result of not enough absorption for the assigned aerosol model.By itself, the relatively bright surface conditions would lead to overestimation of AOD.
We find it interesting that the AOD at Bonanza Creek (64 • N, 148 • W) is severely overestimated in polluted conditions, which suggests that our assumed aerosol model (SSA ∼ 0.91) is too absorbing to represent the dense smoke (SSA ∼ 0.97) observed at the site (Eck et al., 2009), possibly due in part to significant burning of peat fuels in the region.
In general, except for sites that experience widely variable seasonal vegetation states such as Kanpur or Mongu, the characteristics of the MODIS retrieval quality do not vary much from season to season, so the summer month map provides the general global picture.Note that only a few sites meet the minimum ten collocations in heavy loading conditions.

Systematic errors
From Sect. 4 and the literature, we know that significant retrieval biases are tied to particular locations.Many authors have found ways to improve MODIS retrievals at particular sites (e.g., Mi et al., 2007;Oo et al., 2010;Jethva et al., 2008), but we have not implemented them in the global algorithm.Others have empirically "corrected" the MODIS data in poorly performing areas (e.g., Zhang and Reid, for data assimilation purposes), but were not focused on getting to the root causes of problems.Here we determine whether there are residual errors due to specific conditions, including cloud fraction, assumed surface type characteristics, or geometry.Of course, there can be multiple reasons for poorer than average retrieval in a particular scene.For example, coarsedominated dust aerosol type will be more common over more arid, brighter surfaces.Both characteristics would reduce the quality of the retrieved AOD.

Ångstrom exponent
Figure 3 indicated that we have correctly defined the EE (Eq.2) of the MODIS AOD for the global aggregate, however Figs. 4 and 5 show that the accuracy varies by location.What conditions are related to the quality of the comparison? Figure 6 plots the differences between MODIS and AERONET as a function of the AERONET AE, defined by AERONET AOD interpolated to the MODIS wavelengths of 0.47 µm and 0.65 µm, as described in Sect.3.3.Like Fig. 3, the data are sorted and grouped into 50 equally populated bins.Each bin's box represents the statistics of the MODIS-AERONET differences in that bin, with the attributes (except for the whiskers) being the same as described in Fig. 3. Here, the dashed curves represent the over-land EE envelope for total AOD (±(0.05 + 0.15τ ); Eq. 2), where EE is calculated based on the mean AERONET AOD within the AE bin (diamonds; right axis).From this plot, one can assess the average absolute error (and sign), the relative error, as well as the average error in comparison to EE that varies with AE.One also can see whether 66% of the collocations fall within EE for a given bin, and how the absolute and relative errors compare with ER.The EE envelope is larger for coarse-dominated cases, indicating that our sample of coarse-dominated aerosol cases (presumably dust) has larger AOD than our sample of fine-dominated cases.
Figure 6 shows that for the MODIS/AERONET collocations with QAC = 3, MODIS-retrieved AOD is generally accurate where AE is within the algorithm's assumed AE parameter space (0.8 < α < 1.6).Within this AE interval, there is very little variability of AERONET AOD, such that each bin's average AOD is approximately 0.2, with EE of ±0.08.The absolute and relative errors, as well as the ER are all close to zero.For the bins with AE > 1.6 (finedominated), AERONET observed-AOD is lower, so that the corresponding EE has a smaller envelope.Yet MODIS tends to overestimate by ∼0.02 (relative error of 20-30%), which in EE-space is ER ∼0.3-0.4.For coarse-dominated aerosol (α < 0.6) scenes, where the AERONET AOD is generally larger, MODIS tends to underestimate AOD by 0.03-0.04.While this is a somewhat smaller relative error of 15-20%, in EE-space, the average ER is similar in magnitude.
Let us further study the dependence of AOD error and AE, by separating the 58 526 cases into three groups based on AERONET observed AOD. Figure 7 plots the 33 794 cases of light aerosol loading (τ < 0.15) in the top panel, and the 6621 cases of heavy loading (τ > 0.4) in the bottom panel.The remaining cases, having moderate aerosol loadings, are not plotted.For low aerosol loading, the MODIS retrieval of AOD has negligible bias on average, and >66% are within EE, regardless of the scene's AE.There is a small, but systematic MODIS overestimation (∼0.01) for the highest AE cases (α > 2.0).On the other hand, MODIS retrieval of high AOD (τ > 0.4) can have significant errors, especially for the lowest AE cases (α < 0.8).For these coarse-dominated conditions, MODIS underestimates AOD by 0.2 or 20%, lead-ing to poor retrievals compared to EE (ER ∼ 1.0).MODIS underestimation is largest in heavy, dusty conditions.In heavy fine-dominated situations the tendency is towards an overestimation of ∼0.05 or 6%.

Cloud fraction
Although the MODIS cloud-clearing algorithm aims to remove clouds from the scene, many studies have reported a positive correlation between AOD retrieval error and cloud fraction, suggesting residual cloud contamination in the MODIS aerosol retrieval (Kaufman et al., 2005;Marshak et al., 2006;Zhang et al., 2005).The MODIS-aerosol-product cloud fraction is calculated from the 500 m resolution pixels (Levy et al., 2009) that were removed during the cloud masking of the aerosol algorithm (Martins et al., 2002).Figure 8 plots the MODIS-AERONET differences as a function of MODIS-aerosol-product cloud fraction over land.The great majority of cases have low cloud fraction (<5%) and there is no significant bias, such that validation (within global EE) is achieved for scenes with cloud fraction less than 5%.Yet, Fig. 8 also shows that MODIS overestimates AOD when cloud fraction is higher, and the error increases with increasing cloud fraction (e.g., Coakley et al., 2005).As cloud fraction goes above 20%, the mean MODIS error approaches 0.03-0.04 or 15-20% in relative AOD units.Average ER ∼ 0.3-0.4.For the larger cloud fractions (>13%), fewer than 66% are within EE.
The collocation data set that produced Fig. 8 is inherently biased towards low cloud fraction because of the requirement that both AERONET and MODIS report aerosol retrievals at the same time.Globally, the MODIS retrieval will encounter higher cloud fractions than seen in this data set.Thus, the biases seen in Fig. 8 for cloud fraction above 20% will have a greater effect on aerosol statistics calculated from the MODIS retrieval than is apparent from the figure, though "ground-truth" data to assess this situation statistically is lacking.
There are many reasons for AOD dependence on cloud fraction, as well as many possible factors that could increase retrieval error with cloud fraction, such as cloud contamination in the retrieved product.The differences between MODIS and AERONET, however, might not be due entirely to MODIS cloud screening blunders.In fact, they can arise from different strategies for sampling.The AERONET sun mode's cloud screening algorithm tests temporal variability (e.g., Smirnov et al., 2002), whereas MODIS's cloud screening algorithm operates on spatial variability.It is easy to visualize situations where the sunphotometer's view of the sun is unobstructed, yet there are clouds in the surroundings.The AERONET view will be biased towards the clear sky, whereas the MODIS view will include some pixels that are sampled within cloud fields.We know that non-cloudy holes within cloud fields are physically different from the non-cloudy atmosphere far from clouds (Charlson et al.,Fig. 8. Differences between MODIS and AERONET-reported AOD at 0.55 µm (MODIS-AERONET) vs. MODIS-retrieved cloud fraction, for QAC = 3. Explanation of symbols is same as for Fig. 6.
2007; Koren et al., 2007Koren et al., , 2009)).Higher humidity in cloud fields contribute to aerosol swelling close to the clouds (e.g., Twohy et al., 2009), and stray light from 3-D effects (e.g., Wen et al., 2007), remnants of decaying clouds and other cloud-related issues (e.g., Koren et al., 2009) all contribute to increasing the AOD retrieved in the cloud field.Note that some factors that enhance satellite-retrieved AOD in cloud fields should be included in the result (swelling) whereas others represent AOD-retrieval artifacts (3-D effects), but all are physical phenomena that cannot be avoided by cloud masking, unless aerosol retrievals are excluded over the entire cloud field.The paradigm that MODIS does not avoid cloud fields as strictly as AERONET, contributes to the MODIS-AERONET differences in Fig. 8.
We split the cloud fraction cases into three groups based on the AERONET AOD, including those with light (τ < 0.15) and heavy (τ > 0.4) aerosol loadings.We do not plot the results here, but for the light loading cases, the residual cloud fraction bias continues even to lower cloud fraction cases (>2%).For the heavy loading cases on the other hand, the differences between MODIS and AERONET are nearly independent of cloud fraction.This suggests that enhanced AOD associated with cloud fields saturates for higher aerosol loading situations, or that heavy aerosol is not significantly increased in cloud fields.We must remember that although the MODIS validation is constrained by AERONET data and its cloud screening, MODIS may be retrieving in different conditions than AERONET is observing.This means that AOD cannot be easily validated in the vicinity of clouds.

Scene and surface reflectance properties
The MODIS second-generation algorithm makes two major assumptions about the surface optical characteristics.The full inversion expects the scene to be "dark" (observed re-flectance at 2.1 µm must be less than 0.25), and that there are constraints on surface spectral reflectance properties.Specifically, it is assumed that there is a relationship between the visible (VIS: 0.47, 0.65 µm) and shortwave-infrared (SWIR: 2.1 µm) surface reflectance, that also depends on scattering angle and surface "greenness" (Levy et al., 2007b).The surface greenness, parameterized by the NDVI swir (Karneili et al., 2002), is similar to the standard Normalized Difference Vegetation Index (Tucker et al., 1979), but based on two SWIR channels (1.6 and 2.1 µm).Except for extremely dusty cases, the use of the SWIR channels was expected to help minimize aerosol contamination.When developing the C005 algorithm, Levy et al. (2007b) relied on a MODIS granule test-bed, which resulted in a decision to optimize the retrieval to cases with scene reflectance between 0.01 and 0.25, and the NDVI swir dependency to between 0.25 and 0.6.This test-bed was dominated by data from the US East Coast and Western Europe, where the observed 2.1 µm reflectance is ∼0.10, and the NDVI swir is ∼0.4.Although there were scenes that demonstrated a larger range of surface conditions in the testbed, their influence on the global surface parameterization was small.
Figure 5 (top) showed locations over the globe where MODIS and AERONET do not agree in light loading conditions.We can examine the impact of scene and surface assumptions by concentrating on these cases.Figure 9 plots MODIS-AERONET differences in light loading conditions (τ < 0.15), as a function of MODIS-observed scene brightness (reflectance in 2.1 µm; top) and scene greenness (NDVI swir; bottom).Differences between MODIS and AERONET are smallest in the mid-range, when the scene reflectance is 0.12, and/or the NDVI swir is 0.4.MODIS is biased high (by 0.02 or 20%) when the scene reflectance is >0.17 and biased low by similar amount when the scene reflectance is <0.07.The scene's NDVI swir demonstrates a larger influence on the MODIS bias, such that errors are >0.03(30%) when NDVI swir < 0.2 and < −0.03 (30%) when NDVI swir > 0.6.However, even though there are systematic biases, there are only a few conditions for which <66% of the MODIS/AERONET collocations match within EE.These occur when scene reflectance is >0.20 and/or NDVI swir < 0.2, which represents less than 10% of the global dataset.In other words, MODIS tends to overestimate AOD over surfaces that are brighter and less green than optimal, and to underestimate when they are darker and greener.However, for the most part, over the middle of the range optimal for MODIS retrieval, there is very little systematic bias.In order to make MODIS retrieval more accurate over the entire range of surfaces, the assumptions of surface reflectance relationships, surface darkness and surface greenness will be reevaluated for future MODIS retrievals.From Fig. 9, it appears that a simple linear factor could correct most of the observed bias.

Observation geometry
An ideal aerosol algorithm would retrieve AOD of equal quality, independent of solar and observing geometry.However, factors other than algorithm performance can also cause covariance between MODIS AOD retrieval error and observing geometry.For example, many heavy aerosol events (dust, smoke, pollution) occur in mid-latitude and tropical regions during summer.These events tend to coincide with specific scattering geometry.The solar zenith angle (θ 0 ) is small in these circumstances, and scattering angle, is related to solar zenith angle as well as target view zenith θ and relative solar/sensor azimuth φ angles: = cos −1 (−cosθ 0 cosθ + sinθ 0 sinθ cosφ). (4) These factors create natural correlations between observed AOD and scattering angle.Because absolute AOD error increases with AOD (e.g., Fig. 3), correlations between absolute AOD error and scattering angle also occur.However, the relative or fractional error is much less dependent on AOD, so this metric should be relatively independent of geometry.
Figure 10 presents the statistics of the MODIS-AERONET differences as a function of sensor view zenith angle.Angles are negative or positive, depending on whether they are to the "left" or "right" of nadir along the path of the orbit.For example, the left side of the orbit corresponds to the eastern side for Terra (descending across equator), and the western side for Aqua (ascending).The sun is on the western side for Terra and eastern side for Aqua, so there is symmetry to glint and hotspot patterns.
We see from Fig. 10 that although in general, >66% of MODIS-AERONET collocations are within EE, MODIS tends to overestimate AOD by ∼0.01 (5% relative error) on the sun-glint (left) side and to underestimate by similar magnitude on the sun-shadow (right) side of the swath.If split into light (τ < 0.15) and heavy (τ > 0.4) aerosol loadings (not shown), (a) the view angle dependence is limited to low AOD conditions, and (b) the errors are independent of angle in heavy aerosol conditions.Yet, 66% of the collocations fall within the EE envelope, indicating that the EE is an accurate assessment, independent of view zenith angle.
Scattering angle dependence is more difficult to interpret, and is presented for the global aggregate in Fig. 11.Again, the average absolute errors of MODIS are small (<0.01) across the entire range of scattering angle, and for the most part, >66% of collocations in every bin match within EE.However, observed AERONET AOD also increases with scattering angle in three discrete groups: low AOD (<0.15) for smallest angles, medium AOD (∼0.2) for angles between 100 • -140 • and largest AOD (>0.25) for largest angles (>160 • ).Because what we are reporting is based on MODIS/AERONET collocations, the true AOD is correlated with where MODIS is sampling with which angles.High values for AOD are more common in the tropics than near the poles, and conditions of near-nadir solar zenith angles and larger scattering angles are also more common in the tropics than near the poles.This means that larger AOD is associated with larger scattering angle, which is seen in Fig. 11.This can also explain why Fig. 1 of Mishchenko et al. (2009) displays scattering angle dependencies.MODIS tends to overestimate AOD near 120 • , and slightly underestimate between 140 • and 160 • .
The magnitudes of the high and low MODIS biases in these plots are generally negligible, but to avoid misinterpretation due to possible cancellation of errors, we again separate our collocations into light (τ < 0.15) and heavy (τ > 0.4) aerosol loading cases (Fig. 12).For the light loading cases, the angular dependence of the AERONET AOD is reduced, but the pattern of MODIS-AERONET differences is retained from Fig. 11.For the heavy aerosol cases, although in general the relative bias of MODIS is low, there is large negative bias (0.08 or 10%) in the 140 • -160 • angle range.The AERONET AOD is higher (∼0.8) in this range than for smaller angles (∼0.7).We slice our dataset once more, this time separating the heavy aerosol loading cases (τ > 0.4) by AERONETobserved AE.Fig. 13 displays MODIS-AERONET errors for cases of low AE (α < 0.8; top) and high AE (α > 1.2; bottom).The statistics are too sparse to make conclusions about the angular dependence, however, it is clear that there are compensating errors from different aerosol regimes.The presumably dust cases (α < 0.8) are generally underestimated by 0.1 (15-20%), with largest bias of 0.2 (∼25%) in the 140 • -160 • range.For the high AE cases, presumably dominated by fine-mode aerosol, MODIS consistently overestimates AOD, especially in the range 120 • -130 • where the Fig. 12. Differences between MODIS and AERONET-reported AOD at 0.55 µm (MODIS-AERONET) vs. MODIS-observed scattering angle, for QAC = 3, for the "light" (τ < 0.15; top) and "heavy" (τ > 0.4; bottom) aerosol loading cases.The explanation of symbols is given in Fig. 6. bias is ∼0.1 (15%).This points to possible issues with the assumed particle scattering phase functions for both coarse and fine modes.
Finally, returning to the light loading dataset (τ < 0.15), we assess only the cases where 0.3 ≤ NDVI swir ≤ 0.4, where we expect minimal bias due to the surface (e.g., Fig. 9).We plot the scattering angle dependence of these 7510 collocations as Fig. 14, and see that the angular pattern of Figs. 11 and 12 (top) remains.Because the AOD is so small, the angular dependence suggests residual BRF (bidirectional reflectance function) dependence in the surface properties that are not captured in the retrieval assumptions (e.g., Gatebe et al., 2001).In fact, when we constrain to only cases where AERONET-measured AOD < 0.1, the pattern still remains.13.Differences between MODIS and AERONET-reported AOD at 0.55 µm (MODIS-AERONET) vs. MODIS-observed scattering angle, for QAC = 3, for the "heavy" (τ > 0.4) aerosol loading cases only, now separated by AERONET-reported Ångstrom exponent.The top/bottom panels represent cases with "large" (α < 0.8) and "small" (α > 1.8) values of AE.The explanation of symbols is given in Fig. 6.

Terra vs. Aqua
Until now, the validation effort in this paper has been based on the union of the Terra and Aqua collocations.For C004, Remer et al. (2005) compared Terra and Aqua AOD data separately to AERONET, and found no significant differences between their uncertainties.During the development of C005, Levy et al. (2007a) performed preliminary evaluation for the C005 algorithms (using C004 radiance as inputs) and reached the same conclusion.Here, we separate C005 MODIS AOD products into Terra and Aqua cases (QAC = 3), and separately compare with AERONET in more detail.For common dates (after July 2002), the results are shown in Table 3, and there is no significant indication that one instrument compares better to AERONET than the other over land.It is interesting that there is some suggestion that collocated AOD is higher in the morning (Terra; τ ∼ 0.205) Fig. 14.Differences between MODIS and AERONET -reported AOD at 0.55 µm (MODIS-AERONET) vs. MODIS-observed scattering angle, for QAC = 3, for "light"(τ < 0.15) aerosol loading cases only, for a small subset of the NDVI swir range (0.30 ≤ NDVI swir ≤ 0.40).The explanation of symbols is given in Fig. 6.
than in the afternoon (Aqua; τ ∼ 0.195), but considering there are 20% more collocations for Terra than Aqua, there may be some sampling issues; both instruments include some missing dates.

Validation time series
In recent years, there has been some effort to use satellite data to examine global and regional aerosol trends (e.g., Mishchenko et al., 2007;Karneili et al., 2009;Papadimas et al., 2008).In these studies, the magnitude of trends is on the order of 0.01-0.02per decade.We can identify similar trends www.atmos-chem-phys.net/10/10399/2010/Atmos.Chem.Phys., 10, 10399-10420, 2010 Even with known sensor degradation, the MODIS channels used in the aerosol retrieval are maintained by the MCST to within 2% of typical reflectance levels (for example, 0.002 of 0.1 reflectance units) (Xiong et al., 2005).Sensitivity tests with the MODIS algorithm (e.g., Levy et al., 2007b) suggest that such small calibration errors can still result in AOD errors of ∼0.01-0.02.For C005-derived AOD over ocean, Remer et al. (2008) reported a rather large offset of ∼0.015 (10%) between the global means of Terra and Aqua, and suggested the discrepancy was due to calibration differences between the two sensors.For the same study, they concluded there was no significant difference between Terra and Aqua over land.However, if one zooms in on the plots, there is some indication that Terra's AOD is decreasing, whereas Aqua's is increasing over time.
From the time-aggregated validation exercises we have performed so far, we do not see significant differences between Terra and Aqua "quality".However, our aggregation over the entire time series may not reveal a systematic change in quality for one or both sensors that results in canceling errors.If not characterized properly, such systematic change might appear as an artificial global AOD trend.
Let us compare Terra and Aqua separately to AERONET, but as a function of time, and for complete years only.For this purpose, we require AERONET sites that are long-term, and assume that their post-processing (to Level 2) removes artificial trends in the AERONET data time series.Table 4 lists selected AERONET sites with a seven-year or longer record.
Figure 15 plots the Error Ratio (ER) for 0.55 µm AOD (Eq. 3) calculated for every MODIS/AERONET collocation in our multi-station, seven-year record.We see no trend for Aqua, yet a downward trend for Terra.The trend in ER for Terra is statistically significant, as measured by a T-test with 6512 points and correlation of R = 0.215.Terra-MODIS seems to be biased high by 0.005 (∼5%) early in the mission, flipping to a low bias of similar magnitude sometime after 2004.Although, within the validated EE envelope, this suggests an artificial drift in Terra's AOD time series.
To explain the drift, we recall that the over-land retrieval inverts reflectance in three channels (0.47, 0.65 and 2.1 µm).The 0.47 µm channel, like the other MODIS blue and deep blue channels at 0.412 and 0.443 µm, is affected by polarization and directional signal issues.This is especially true for Terra, which suffers from more significant optical sensor degradation than does Aqua (X.Xiong, personal communication, 2009).However, unlike the 0.412 and 0.443 µm channels, which are closely monitored by the ocean color team (http://oceancolor.gsfc.nasa.gov/VALIDATION/operational gains.html)and tuned for the bio-optical retrieval algorithms, the 0.47 µm channel may have residual calibration error.Sensitivity tests show that a systematic change in the 0.47 µm channel, capable of causing the trend seen in MODIS-Terra's ER record, is entirely possible.Preliminary analysis of the 0.47 µm reflectance suggests that there is such a residual time-dependent trend.We can see how the process of MODIS validation can help reveal hidden biases or uncertainties in the calibration algorithms.Certainly, calibration has been an important consideration in other data validation efforts (e.g., Lyapustin et al., 2007;Kahn et al., 2005;Lallart et al., 2008).Careful reevaluation of the Terra/MODIS 0.47 µm channel is ongoing, and refinements will be made, as appropriate.
Of 85 463 valid MODIS/AERONET collocations (at 0.55 µm), 68.8% of matching AOD values fell within the expected error (EE) bounds of ±(0.05 + 15%), a criterion for MODIS-C005 AOD product validation.When separated by QAC, only 50% of the collocations having low confidence (QAC = 0; N = 10743) matched within EE, whereas 72% of those with high confidence (QAC = 3; N = 58726) matched within EE.Those collocations having QAC = 3 also demonstrated regression fits extremely close to 1-1.This means that stratifying by QAC can be significant for some applications, and suggests that whenever possible, users should rely on the highest confidence data for quantitative studies.The use of lower confidence data should depend on the tradeoff between an application's tolerance for uncertainty and the spatial coverage requirements.
Even when constrained to the highest confidence data, comparison of MODIS-derived Ångstrom Exponent and fine AOD showed that MODIS does not provide quantitative in-formation about aerosol size over land.Thus, we strongly recommend that users NOT use size products quantitatively.To avoid confusion, we plan to remove AE and fine AOD parameters from future product lists.The ETA (model weighting) parameter will be retained for its diagnostic value.
Although MODIS Collection 5 AOD uncertainties have been characterized globally, and the product is therefore validated, the literature clearly points out regional and systematic errors.Many of these errors can be traced to situations where assumptions about surface reflectance and/or assigned aerosol optical properties are violated.
By considering only cases having light aerosol loading (τ < 0.15, N = 33794), we focused on issues related to the assumed surface reflectance.We characterized the MODIS-AERONET differences as functions of observed NDVI swir and 2.1 µm scene reflectance.
-MODIS underestimates AOD (by 0.02 or more) where the surface is unusually dark (2.1 µm reflectance < 0.05) or green (NDVI swir > 0.6).These conditions are seen in parts of the Amazon forest, as well as northern Australia and areas known for reddish colored soils.
By considering only cases of heavy aerosol loading (τ > 0.4; N = 6621), we focused on issues related to assumed aerosol type properties.
-MODIS-derived AOD compares best to AERONET where the observed aerosol is fine-mode dominated, and the algorithm-assigned model's single scattering albedo is appropriate.These conditions are met over the East Coast of the US, Western Europe, and parts of Asia.
-MODIS overestimates in fine-dominated cases where the observed SSA is greater than that assumed.For example, for the Brazilian Cerrado, the assumed SSA = 0.86 whereas more recent AERONET data suggests SSA = 0.91.
-MODIS underestimates in cases where observed SSA is less than that assumed for the fine-mode model, and/or that dust is mixed in.This is noted in southern Africa, and along the Sahel and Indian semi-arid zones.
As a result of these findings, we suggest that maps of aerosol model type assignments be modified for future versions of the aerosol algorithm.
Although they are clearly tied to specific regions, systematic biases were found to be dependent on a variety of observed and assumed conditions.In general, MODIS underestimates AOD for low AE (α < 0.8), and overestimates AOD for high AE (α > 1.6), as measured by AERONET.The low AE cases tend to be characterized by larger total AOD, and the underestimation is more significant for heavy aerosol loading.Clearly, the MODIS algorithm does not work well for dust.There is no systematic bias for the middle range of AE (0.8 < α < 1.6), which suggests that the MODIS Collection 5 algorithm performs appropriately when the ambient conditions are similar to the algorithm's expected AE range.
The MODIS bias is positively correlated with observed cloud fraction.As cloud fraction increases from 10% to 60%, the average MODIS-AERONET differences increase to 0.03 (∼15% relative error).The MODIS bias is not significantly correlated with cloud fraction for conditions of heavy aerosol loading.Yet, we note that AOD cannot easily be validated in the vicinity of clouds, because the sampling of cloud and cloud free sky may be different for MODIS and AERONET.
The MODIS-AERONET differences are also correlated with scattering angle.However, part of this dependence is clearly related to MODIS's sampling dependence on scattering angle.Dust and absorbing aerosol types dominate tropical regions, which coincidently are the only regions that MODIS can observe with large scattering angle.Studying the retrieved results stratified by particular aerosol/scene conditions can help to isolate true artifacts in the retrieval.For example, when constrained to cases of light aerosol loading (τ < 0.15) and optimal surface vegetation conditions (e.g., 0.30 ≤ NDVI swir ≤ 0.40), we observe that MODIS-AERONET differences exhibit some dependence on scattering angle.This suggests that our assumed surface reflectance may be missing some BRF factor, and should be studied further.
We find no significant differences in retrieval biases between Terra and Aqua.However, there is a statistically significant change in the Terra-MODIS/AERONET comparison.Terra-MODIS tends to overestimate AOD by <0.01 before 2004, and to underestimate AOD by similar magnitude thereafter.The likely cause is degradation of Terra's optical response in the 0.47 µm channel (used in the land algorithm only) that results in very small errors to the sensor's calibration over time.This effect is not found with Aqua data.The calibration issues should be corrected in a future reprocessing of MODIS data, and must be completely resolved before attempting to derive global trends from the data (e.g., Zhang and Reid, 2010).
In this paper, we performed overall assessment of the MODIS AOD relative to AERONET, and validated it globally.Despite systematic biases due to AE, cloud fraction and surface scene conditions, nearly 80% of the retrievals fall into a parameter range where AOD errors were less than 0.01.Thus, the MODIS AOD is a quantitative product, and is useful for applications such as model assimilation (e.g., Hyer et al., 2010).The biases discussed above and displayed in the figures should be acknowledged and considered when using MODIS land AOD retrievals in cloudy scenes, in heavy dust or smoke, over dense vegetation or bare soils.Alternatively, these situations could be avoided.
Here, we also assessed the performance of the MODIS aerosol (AOD and size parameters) compared to AERONET observations.We have not attempted to characterize MODIS data that is not collocated with AERONET.However, we now have defined a quantitative EE for the MODIS Collection 5 dark target aerosol products over land.We encourage users to take the known biases into account as they use the Collection 005 products.The results presented here will provide a solid base from which to refine the algorithm and prepare for future Collections of the MODIS products.

Fig. 1 .
Fig.1.MODIS C005 AOD at 0.55 µm over dark land (QAC ≥ 0) collocated with AERONET (quadratically interpolated) to the same wavelength, for both (the union of) Terra and Aqua datasets.Data are sorted according to ordered pairs (AERONET, MODIS) of AOD in 0.025 intervals, so that color represents the number of cases (color bar) having that particular ordered pair value.The dashed, dotted and solid lines are the 1-1 line, EE for land AOD ±(0.05 + 0.15), and the linear regression of the pre-sorted scatterplot, respectively.Text at the top describes: the number of collocations (N), the percent within expected error, the regression curve, correlation (R), and the RMS error of the fit.Note that each axis ranges from −0.1 to 3.0, to show allowed negative values.

Fig. 2 .
Fig. 2. MODIS-derived 0.47/0.65 µm AE (top) and 0.55 µm fAOD (bottom) collocated with analogous AERONET parameters, for both (the union of) Terra and Aqua datasets.Data are sorted according to ordered pairs (AERONET, MODIS) of AOD in 0.025 intervals, so that color represents the number of cases (color bars) having that particular ordered pair value.For each panel, the dashed, dotted and solid lines are the 1-1 line, EE, and the linear regression of the pre-sorted scatterplot, respectively.Text at the top describes the number of collocations (N ), the percent within expected error, the regression curve, correlation (R), and the RMS error of the fit.The AERONET AE is derived from AOD that has been interpolated (quadratic) to MODIS wavelengths, whereas the AERONET fAOD was derived with the ONeill et al. (2003) deconvolution technique.EE for AE is ±0.4,EE for fAOD is ±(0.05 + 0.20).

Fig. 3 .
Fig. 3.Absolute error of AOD (MODIS-AERONET) at 0.55 µm vs. AERONET-derived AOD at 0.55 µm, for QAC = 3.The x-axis is the AERONET derived AOD, and the y-axis is the absolute MODIS-AERONET AOD difference.Data are sorted by AERONET AOD and grouped into 50 equal bins.Each boxplot represents the statistics of the MODIS-AERONET differences in the bin.The means and standard deviations of the AERONET AOD are the centers and half widths in the horizontal (red).The mean, medians, and 66% (1-σ ) interval of the MODIS-AERONET differences are the black squares, the center, and top-bottom red intervals in the vertical (also red).The blue whiskers are the 96% (2-σ ) intervals.The red dashed curves are linear best fits to the 66whereas the green dashed curves represent the over-land EE for total AOD ±(0.05 + 0.15).

Fig. 5 .
Fig. 5. "Quality" of MODIS/AERONET 0.55 µm AOD comparison at each AERONET site during June-July-August, for cases where AERONET AOD < 0.15 (top) and AERONET AOD > 0.4 (bottom).Sites are color-coded based on the average error ratio (Error/EE).Greenish colors (Cyan and Lime) are sites where the average of the MODIS and AERONET-derived AOD values differ by less than half of EE.Colder (warmer) colors represent cases where MODIS, on average, significantly underestimates (overestimates) AOD.

Fig. 6 .
Fig. 6.Differences between MODIS and AERONET-reported AOD at 0.55 µm (MODIS-AERONET) vs. AERONET-observed AE, for QAC = 3.Data are sorted by the AE and grouped into 50 equal bins.Each box represents the statistics of the MODIS-AERONET differences in the bin.The means and standard deviations of the AE (each bin) are the centers and half widths in the horizontal.The mean, medians, and 66% (1-σ ) interval of the MODIS-AERONET differences are the squares, the center, and top-bottom intervals of the boxes.The dashed curves represent the over-land EE envelope for total AOD ±(0.05 + 0.15), where EE is calculated based on the mean AERONET AOD within the bin (diamonds; right y-axis scale).

Fig. 10 .
Fig. 10.Differences between MODIS and AERONET-reported AOD at 0.55 µm (MODIS-AERONET) vs. MODIS-observed sensor zenith angle, for QAC = 3 over land.Explanation of symbols is same as for Fig. 6.Note that the negative values of sensor zenith angle refer to the "left" of the MODIS swath along the track (West side for Aqua, East side for Terra).

Fig. 15 .
Fig. 15.Time series of Error Ratio (ER; Eq. 3) of MODIS C005 0.55 µm AOD compared to seven long-term AERONET sites, for Terra (top) and Aqua (bottom).Points between the dashed lines (±1) are cases where MODIS matches AERONET within EE over land (±0.05 ± 0.15τ ).The solid line is the linear regression.At the top of the plot is text that describes: the number of collocations (N), the regression equation and correlation (R).

Table 1 .
Statistics of the comparison between MODIS and AERONET total spectral AOD over land, N = 85463.

Table 2 .
Statistics of the comparison between MODIS and AERONET total AOD at 0.55 µm over land, as a function of QAC.

Table 3 .
AERONET sites with long-term records used for time series assessment.

Table 4 .
Statistics of the comparison between MODIS and AERONET total AOD at 0.55 µm over land, (QAC = 3) for Terra and Aqua since July 2002.