Aerosols are hypothesised to cool the climate system due to their ability to enhance the reflection of clouds , particularly ubiquitous warm-phase clouds located in the boundary layer . The uncertainty attached to this cooling effect is considered to be very large in both satellite and general circulation model (GCM) estimates. The fundamental issues causing this large uncertainty stem from a myriad of challenges related to retrieval artefacts in satellite products (e.g. cloud contamination and 3-D radiative effects) and missing processes in GCMs (e.g. parameterisation schemes related to cloud-top entrainment feedbacks and buffering in aerosol–cloud interactions as discussed in ). Satellite observations suggest that the aerosol indirect radiative cooling effect is about half that of GCM-based estimates (e.g. see Fig. 7.19 in chap. 7 of the IPCC 5th Assessment Report, ). For future climate projections it is critical to close the satellite–GCM gap in the forcing estimate so we can understand to what extent anthropogenic aerosols may have cooled, and continue to cool, the climate system.

Despite the advances in satellite-based retrievals in recent decades, obtaining robust statistical relationships between aerosols and clouds remains difficult. Challenges associated with obtaining accurate passive satellite retrievals generally involve the following issues: (1) an artificially high aerosol optical depth (AOD) retrieval due to the presence of undetected cloud in areas assumed to be cloud-free (cloud contamination) ; (2) radiation scattered by clouds that illuminate the aerosol field and are not accounted for in the 1-D radiative transfer model in satellite retrievals causing erroneously high AOD retrievals (3-D effects) , and (3) humidification causing aerosols to swell near clouds, thereby enhancing AOD without any increase in aerosol number concentration (humidification effect) . Attributing the contribution of these mechanisms to the enhancement in AOD and particle size near clouds is difficult using passive sensing instruments from satellite-based observations alone.

Recent assessments characterising near-cloud aerosol retrieval artefacts were examined using space-borne lidar and ground-based lidar observations. In general, the Moderate Resolution Imaging Spectroradiometer (MODIS) aerosol products show enhanced aerosol optical depth and particle size several kilometres beyond the cloud edge . Evidence from selected Surface Radiation Budget Network (SURFRAD) observations reported in suggest that near-cloud contamination in the satellite retrieval can enhance the aerosol cloud cover fraction relationship by approximately 40 % (i.e. cloud cover fraction is larger for the same aerosol optical depth when cloud contaminated pixels are used in the analysis). In addition, the direct aerosol radiative effect (i.e. the increase in reflected radiation due to aerosol loading in clear-sky conditions) in satellite observations is significantly larger (35–65 %) than that inferred from larger (greater than 20 km2) cloud-free ocean regions where the aerosol retrievals are located farther from the clouds . The extent to which these near-cloud aerosols influence the correlation-based statistics used as diagnostics for aerosol–cloud interactions in satellite observations is largely unknown at the global scale. The quantification of their impacts on the aerosol radiative forcing estimate is the goal of this study.

Besides the errors in the satellite retrieval, attributing causal relationships between non-collocated cloud and aerosol retrievals is challenging. Passive sensors cannot currently retrieve aerosol and cloud simultaneously, because the imager pixels are classified as being either “cloud” or “cloud-free”. To solve the collocation problem a pre-averaging methodology is commonly used e.g.. Typically, the aerosol and cloud properties are pre-averaged over broad regions as a means to encapsulate both retrieval types within a (level-3-type) grid box. The aerosol and cloud bins are typically constructed at a spatial scale of 1∘×1∘ at daily time intervals. This method limits the number of sampling pairs to at most 90 (for a 3-month period) from which seasonal regression statistics can be derived. Some studies have also used a back-trajectory model to pair aerosols and clouds and compute aerosol cloud diagnostics using the assumption that the aerosol properties remain constant over the coarse of the trajectory. A disadvantage to using these approaches is if the artefacts in the pixel-scale retrieved data (i.e. the level-2 data) near clouds are not properly screened, the aerosol–cloud relationships may be biased regardless of the methodology used to combine the observations from satellite data.

Another, less common method of collocating aerosols and clouds is data assimilation. This process uses aerosol properties that are extracted from reanalysis products at the location of the observed cloud . However, the aerosols simulated by the models may be strongly affected by wet deposition , particularly in locations where the satellite observations cannot provide a constraint on the aerosol loading in cloudy areas .

In this paper, we have developed a new collocation method, defined here as the high-resolution cloud–aerosol pairing algorithm (CAPA). In this method high-resolution pixel-scale cloud observations are paired to the nearest aerosol retrieval within the satellite image. These pairs are then aggregated over 1∘×1∘ regions. This results in a significant boost to the total number of samples used in the regression statistics from about 90 (using 1∘×1∘ pre-averaged daily statistics) to approximately 11×105 using CAPA over a 3-month period. In addition, the distance between cloud and aerosol retrievals can be adjusted in CAPA as a means to screen for aerosols next to clouds. This method provides the ability to expose biases in the aerosol–cloud radiative forcing estimate due to satellite artefacts in the retrievals of aerosols near-cloud.

The paper is organised into the following format: satellite data sets and their corresponding retrieval algorithms are described in Sect. 2, the procedure to compute aerosol indirect radiative forcing is in Sect. 3, CAPA is described in Sect. 4, compositing techniques are described in Sect. 5, the results using this approach are compared to the standard pre-averaging method as described in Sect. 6, and the summary and discussion of this work and how it relates to our companion paper are described in Sect. 7.

Two passive sensors with similar equator crossing times (approximately 10:30 am local), the Advanced Along-Track Scanning Radiometer (AATSR) on Envisat and MODIS on Terra, are used in this study. AATSR is a dual-view instrument having a footprint resolution of about 1 km at the surface. The two views (satellite zenith angles at ∼55∘ forward and nadir) offered by the instrument provide near-simultaneous (within 90 s) observations of the same location on the Earth. This provides the ability to separate the surface from the atmospheric signals in order to increase the accuracy of the 1 km aerosol retrieval (in version 4.01 the aerosol is reported at 1 km resolution) using the Optimal Retrieval of Aerosol and Cloud (ORAC) algorithm . Cloud properties are also retrieved using the same inputs (e.g. the cloud mask, surface reflectance data over the ocean, basis for the optimal estimation scheme in the radiative transfer model, and thermodynamic profiles) as the aerosol retrieval algorithm. By using the same inputs for the ORAC forward model we achieve a high degree of consistency between the aerosol and cloud products. This consistency is essential for constraining the complexity in attributing cause and effect in aerosol–cloud interaction studies. The ORAC algorithm for cloud is described in and most recent improvements to the forward model code in and . The high-resolution pixel-scale data are currently available through the European Space Agency (ESA) Climate Change Initiative (CCI) although the aerosol products are averaged to a spatial resolution of 10 km to be consistent with other satellite products of this kind. The pre-averaged gridded daily products (L3C) used in this study are also available via http://cci.esa.int.

Shortwave and longwave broadband radiative fluxes are obtained using the CC4CL (Community Cloud Retrieval for Climate) algorithm . The code uses the BUGSrad radiative transfer model in conjunction with the input cloud and aerosol properties derived from ORAC. BUGSrad is based on the two-stream approximation and correlated-k distribution methods of atmospheric radiative transfer. It is applied to a single-column atmosphere for which the cloud and aerosol layers are assumed to be plane-parallel. Cloud and aerosol properties retrieved using ORAC are ingested into BUGSrad to compute both shortwave and longwave radiative fluxes for the top and bottom of the atmosphere. The algorithm uses 18 bands that span the electromagnetic spectrum to compute the broadband flux – 6 in the shortwave and 12 in the longwave. BUGSrad shows good agreement with the Clouds and the Earth's Radiant Energy System (CERES) observations and has been used to assess the Earth's energy budget using CloudSat observations .

The MODIS 1 km pixel-scale data are obtained from the Terra satellite. Terra was selected for comparison, because it has a similar orbit and equator crossing time as Envisat. To obtain top of atmosphere (TOA) radiative fluxes from MODIS on Terra we ingest the standard collection 6 cloud MOD06; and aerosol MOD04; products into the CC4CL radiative flux BUGSrad model. Because the MOD04 product is sampled at 10 km spatial resolution the data have been resampled at 1 km resolution to match the cloud product. The ESA CCI projects evaluated products; among them were the standard collection 6 MODIS aerosol and cloud products. ORAC was also applied to MODIS data (within Cloud_cci) and the MODIS–ORAC product was found to agree very well with the MODIS standard products .

This analysis uses 10 years of AATSR observations spanning from 2002 to 2012. Pixels are screened to include only low-level (cloud top pressure greater than 500 hPa) liquid-phase (cloud top temperature greater than 273 K) maritime clouds. Due to limitations in data storage we are only able process 3 months (June, July, and August; JJA) using high-resolution MODIS–ORAC (broadband fluxes derived using standard MODIS products) Terra retrievals for comparison.

Computation of the aerosol indirect radiative forcing estimate is based on a top-down approach in which a system-wide variable, the cloud radiative effect (CRE), is used to compute the radiative forcing as a function of the aerosol loading. This reduces the number of free parameters to just a few (e.g. cloud fraction, cloud albedo, and aerosol index) in which the observational uncertainties are better known compared to other quantities that are difficult to retrieve from passive satellite measurements (e.g. droplet number concentration, cloud condensation nuclei, and cloud thickness). The derivation of the shortwave component of the aerosol indirect radiative forcing at the top of the atmosphere is the same as that used in and is derived from the cloud radiative effect equation written here as FCRE=Fclr-Fall-sky, where Fclr is the clear-sky net radiative flux (i.e. Fclr=Fclr↑-Fclr↓, arrows denote the upwelling and downwelling fluxes, respectively) for atmospheric columns in containing no clouds and Fall-sky is the net flux that is observed for all-sky conditions (excluding ice clouds here) computed from clear- and cloudy-sky 1 km pixels located within each 1∘×1∘ region. The thermodynamic profiles used in the broadband flux calculations are interpolated to each 1 km imager pixel from the N256 spatial resolution of the ECMWF (European Centre for Medium range Weather Forecasting) interim reanalysis product. Assuming a relatively dark ocean surface Eq. (1) can be decomposed into Fall-sky=(1-cf)Fclr+cfFcld, where Fcld is the component of the radiative flux contributed by clouds and cf is the cloud cover fraction over the observed area. Combining Eqs. (1) and (2) and considering the shortwave component of the upwelling fluxes, the cloud radiative effect becomes FCRE=cf(Aclr-Acld)F↓, where Aclr is the clear-sky albedo, Acld is the albedo of the cloud and F↓ is the daily-mean incoming top of atmosphere solar radiation. Taking the derivative of Eq. (3) with respect to the aerosol index (AI) gives the column TOA cloud radiative effect aerosol sensitivity dFCREswdln⁡AI=cm‾dAclrdln⁡AI-dAclddln⁡AI+(Aclr-Acld)‾dcfdln⁡AIF↓, where cm‾ is the climatology of clouds having cloud top pressure greater than 500 hPa and composed of liquid-phase droplets over ocean regions. The derivative terms represent the change in clear-sky and cloudy-sky albedo as a function of aerosol index. The first term on the right-hand side of Eq. (4) is called the intrinsic aerosol effect and includes the impact of aerosol on changes in cloud albedo. The second term is the extrinsic effect, which represents the impact of aerosol on cloud fraction. For further details regarding the derivation of these terms see .

We use aerosol index (AI =τa× Å, where τa is the aerosol optical depth at 550 nm and Å is the Ångström exponent derived from the optical depths at 550 and 869 nm), because AI has been shown to serve as a better indicator of the column cloud condensation nuclei compared to aerosol optical depth . To limit co-variation between the annual cloud and aerosol cycles with the incoming solar radiation flux, the aerosol–cloud sensitivities are computed, first over each season separately and then combined to form the annual mean aerosol indirect effect sensitivity.

Finally, the aerosol indirect forcing estimate is obtained by multiplying Eq. (4) by an amount of aerosol attributable to anthropogenic activities as determined by MACC-II reanalysis data (i.e. ΔτaMACC=ln⁡(τaMACC)-ln⁡(τaMACC-τanthMACC)=ln⁡11-Fanth, where τanthMACC is the aerosol optical depth attributed to anthropogenic activities). In this calculation we assumed that the fractional change in the anthropogenic AOD is equivalent to the fractional change in anthropogenic AI. This assumption may lead to an underestimation of the aerosol indirect forcing, especially in regions where dust dominates . MACC-II is a satellite-model hybrid data set that utilises the state-of-the-art ECMWF-IFS (Integrated Forecast System) aerosol transport model along with a surface emissions inventory and assimilated MODIS data to provide aerosol optical depth for a variety of species including dust, organic carbon, sea-salt, black carbon, and sulfate (e.g. see and for details). Global distributions of the anthropogenic aerosol fraction are provided in and ; the global oceanic mean value is about 21 %.

The basic approach of CAPA is to pair each cloud observation to the nearest aerosol retrieval. This process is performed within the satellite image (the swath width is 2330 km for MODIS and 512 km for AATSR). Distances using Euclidian geometry in pixel-coordinates are computed between each cloud pixel and all possible aerosol retrievals. Given the large number of pixels within the swath this task is computationally demanding if proper screening is not carried out initially. To decrease computation time and the number of possible pairs (for a given cloud observation), the satellite image is divided into sections and pixels are grouped together (typically 250×250 pixel regions). For pixels near the edge of each square region (within 125 km), the search radius is extended to the nearest adjacent square region. These steps eliminate aerosol–cloud pairs at distances greater than approximately 150 km (for 1 km pixel-scale data), but potential pairs beyond this length scale occur at low frequencies and are probably less relevant since the cloud–aerosol interaction is less likely to be influenced by the same air mass .

In the next step the cloud fraction is computed (using the satellite retrieval cloud mask at 1 km resolution) as a metric to determine the appropriate search algorithm to use for the given subsection of the satellite granule. If the cloud fraction is high (cf>50 %) then fewer aerosol retrievals exist and the distance between each cloud pixel and all aerosol pixels is computed simultaneously via brute force within that region of the satellite orbit. However, if the cloud fraction is low (cf<50 %) the number of aerosol targets will be larger and the former approach will run much more slowly. In this case the pixels that are adjacent to the cloud observation are searched first (as there is a relatively high probability an aerosol is located in an adjacent pixel when the cloud fraction is lower). If an aerosol retrieval is not found in the adjacent pixels this step is repeated again until an aerosol target is found. By using the search algorithms together (based on cloud fraction) the computation speed decreases by more than 50 % compared to running the retrieval in brute-force mode alone. Finally, if two (or more) aerosol pixels are located at the same distance from the cloud observation then one of them is selected at random. The CAPA run time is approximately 2–6 min using a single core Intel/AMD at ∼ 3 GHz processor for a typical MODIS granule that contains 2.5 million 1 km pixels.

The results of CAPA are applied to part of an AATSR orbit and displayed in Fig. . The image shows a belt of clouds across the middle of the granule with aerosols retrieved on both sides (Fig. b). The cloud-belt is approximately 300 km wide (Fig. c). Aerosols in the lower half of the image have lower optical depths and tend to be located at greater distances to the nearest cloud (green pixels), whereas the aerosols in the upper half of the image are retrieved mostly in a broken cloud field in which the aerosols are located close to the clouds (red pixels). The aerosol retrievals have a “blocky” appearance, because the standard level-2 aerosol products from MODIS (MOD04) and AATSR (V4.02) are averaged over larger 10 km2 pixels. In order to match the cloud and aerosol products and form aerosol–cloud pairs (on the same imager pixel grid), the aerosol products for AATSR and MODIS are resampled at 1 km resolution.

Aerosol statistics are examined as a function of distance to the nearest cloud in Fig. . The observations are comprised of 3 months (JJA in 2008) of data collected across four different 10∘×10∘ regions (off the coasts of California, Peru, the Azores, and Namibia). In every region, the mean aerosol optical depth increases, the Ångström exponent decreases and the aerosol index increases as the observations are closer in distance to the nearest cloud. In agreement with previous studies we find that about half of all clear-sky columns occur within 4–5 km of the low-level cloud . As a consequence, near-cloud aerosols (that are potentially affected by retrieval artefacts) in pre-averaged aerosol data sets would provide substantial weight for the aerosol statistics.

Despite efforts to limit contamination in standard MODIS collection 6 retrievals, threshold values for the darkest and brightest 25 % of pixels in each cluster are applied. One complication with the standard MODIS products is that the cloud and aerosol products use different cloud masks to perform their retrievals. Nevertheless, large aerosol optical depths remain in the MODIS-observed pixels near cloud edges, due primarily to 3-D effects and the swelling of aerosols by higher relative humidity. Currently, no effort is made to remove near-cloud pixels from the AATSR ORAC gridded products. This is evident from the faster and less pronounced decrease in aerosol optical depth and Ångström exponent with distance from the nearest cloud in the MODIS product. At distances greater than 15 km the aerosol optical properties tend to be fairly constant. Similarly, also noted that beyond 15 km contamination effects were minimised in MODIS data. Therefore aerosol–cloud relationships are also examined for pairs beyond the 15 km length scale.

Results are based on four distinct composites of the data (listed in Table 1). Each composite contains aerosol and cloud pairs which are aggregated into 1∘×1∘ regions so that regression statistics can be computed on the basis of the linear least-squares slope between the cloud albedo and the aerosol index. This process is also carried out separately for pixels designated as “cloud-free” in order to derive the clear-sky albedo change as a function of aerosol loading (i.e. dAclrdln⁡AI) in Eq. (4). Two distinct methodological frameworks are used: (1) the high-resolution CAPA and (2) the low-resolution pre-averaging methods.

CAPA is run using two distinct length scales: (1) clouds are paired with the nearest located aerosol (CAPA-L2) and with the nearest aerosol that is located at least 15 km away from any other cloud (CAPA-L2_15km). The median distance between aerosol and cloud pairs from these two composites is 8.2 and 27.1 km, respectively, thereby providing the ability to screen for aerosol next to cloud. Another advantage of CAPA is that it retains individual L2 pixels, i.e. full-resolution data, for quantifying aerosol–cloud relationships over each 1∘ grid box, allowing for more of the aerosol field to be sampled and thus providing more degrees of freedom with which to derive seasonal statistics. For example, the average number of high-resolution samples going into a typical 1∘ region over a 3-month period is approximately 3.0×105, although the actual number of degrees of freedom is much smaller (each region provides on average about 10 degrees of freedom per day providing roughly 1000 over a 3-month period) owing to oversampling the same aerosol used to make cloud–aerosol pairs. Nevertheless, the typical number of samples in the CAPA composites far exceed those that can be achieved using the standard pre-averaged products (PRE_AVG-L3 and PRE_AVG-L3_Corr.) where at most they provide 90 samples (providing one aerosol–cloud pair sample per day for a given grid box) in a 3-month period. As pointed out later, the reduced sampling in the PRE_AVG-L3 composite results in a larger 1σ standard error regression error estimate.

Aerosol–cloud relationships are examined using the CAPA and pre-average methods at a variety of spatio-temporal scales. An assessment of CAPA is carried out at both the regional scale (10∘×10∘) in selected locations with predominately low-level clouds (i.e. off the coasts of California, Peru, the Azores, and Namibia) and at the global scale.

Two independent satellite instruments, AATSR on Envisat and MODIS on Terra, were used in this study to test a new cloud–aerosol pairing algorithm to compute aerosol indirect forcing estimates for warm oceanic maritime clouds. Cloud–aerosol pairs formed at the pixel-scale resolution of the satellite imager using CAPA can provide a larger sample size from which to compute correlation statistics between cloud albedo and aerosol index. The effect of the larger sample size effectively decreases the standard error of the regression slope, thereby providing higher confidence in the new radiative forcing estimates.

The scheme was also developed to determine the extent to which artefacts in aerosol retrievals located in the vicinity of clouds affect aerosol–cloud relationships in space-borne satellite observations. By removing aerosols located within 15 km of the nearest cloud, the cloud albedo effect (intrinsic) and cloud fraction effect (extrinsic) forcing decreased by 40 and 70 %, respectively. These new estimates suggest that aerosol effects on the radiative properties of clouds are even smaller than previously demonstrated from satellite-based studies. This new methodology therefore further widens the gap between the satellite and the very strong forcing estimates derived using most GCMs.

One inherent limitation to CAPA is it cannot be used to estimate the extrinsic (or overall) aerosol indirect forcing at the pixel-scale resolution of the satellite imager (typically at 1 km). This is because the pairing algorithm uses pixels that are cloudy to do the pairing, thereby resulting in a grid-box mean cloud fraction value of 1.0. It is also generally not practical to use level-2 pixel-scale data in satellite-based assessments due to the large volume of data that are required at this scale. Therefore, we propose using CAPA (or another analogue) to remove the potentially contaminated aerosols within 15 km from nearby clouds as a first step to constructing pre-averaged level-3-type data sets. We demonstrate using 10 years of AATSR data that the broader-scale level-3 data can be corrected to yield the same intrinsic aerosol–cloud sensitivities using the level-2 pixel scale, thereby providing a means to also compute the extrinsic aerosol indirect radiative forcing using the PRE_AVG broader-scale level-3 data.

This initial version of the CAPA algorithm has been used here to highlight a potential source of error in satellite-based estimates of aerosol indirect forcing. The CAPA method highlights areas where we have trust in the satellite-based retrievals that are pertinent for aerosol–cloud interactions. In subsequent versions of this algorithm additional steps could be pursued to increase computational efficiency. There are more efficient algorithms for finding nearest neighbours in a large data set. Binary search trees, such as a k-dimensional or vantage-point trees, would probably work very well within CAPA. Furthermore, this work would benefit from a deeper examination of the coupling between aerosol and cloud pairs using a back-trajectory model (such as the Hybrid Single Particle Lagrangian Integrated Trajectory) following the method described in .

Extension of this method in comparison with GCMs is explored in the companion paper of . Furthermore, this companion paper also quantifies the impacts of meteorology on the aerosol–cloud relationships using numerous meteorological regimes based on lower-tropospheric stability and free-tropospheric relative humidity. These regimes are used for comparison with the global aerosol–climate model ECHAM6-HAM2. Also, the new CAPA diagnostics are used to select aerosols that are located far from clouds, thereby providing a more consistent comparison to the GCM which simulates interactions using dry-mode aerosols. The influence of dry-mode aerosols are also found to produce a much smaller indirect forcing estimate due to the dry aerosol being a better proxy for cloud condensation nuclei. These aspects are developed and explored further in .