The CAPTIVATE algorithm provides a framework for a variational, or optimal estimation, approach to the inverse retrieval of vertical profiles of rain, ice and snow, liquid cloud, and aerosols from one or more vertically pointing active and passive instruments. CAPTIVATE is novel in that the measurements used and the state variables retrieved are easily configurable so that the same algorithm can be applied to space-borne, airborne, and ground-based measurements. The retrieved state variables and the representation of each class of hydrometeor can also be modified as appropriate. The variational approach allows for a robust treatment of uncertainties in the retrieval subject to the appropriate selection of observational uncertainties, forward model errors, and physical constraints.

Here we present a general description of the CAPTIVATE retrieval; justifications for the settings used in this study are made in later subsections. The retrieval is made for each profile by iterating to find a state vector that minimizes the cost function given by J=12δyTR-1δy+12δxTB-1δx+Jc(x), where δy = y - yf is the difference between the observed (y) and forward-modelled (yf) measurements; R is the error covariance matrix of δy, the sum of the error covariance matrices of the observations and the forward model; δx = x - xa is the difference between the state (x) and its a priori estimate (xa); B is the error covariance matrix of xa in which the diagonal elements are the error variances of x; and Jc(x) provides the capability to apply flatness and smoothness constraints to reduce the effect of observational noise on the state vector . Additionally, profiles of retrieved variables can be represented smoothly as a set of cubic spline basis functions and Sect. . The minimization of the cost function is carried out by iterating on the state vector beginning from the priors in the direction of the first and second derivatives of the cost function the Levenberg–Marquadt method;.

The rain DSD is given by a normalized Gamma function of the form N(D)=NwΓ(4)3.674(3.67+μ)4+μΓ(4+μ)DD0μexp⁡-(3.67+μ)DD0. This formulation is a function of three independent, physically meaningful parameters for the shape μ, median drop size D0, and normalized drop number concentration intercept Nw of the DSD . The shape factor μ is of secondary importance to D0 and Nw in terms of the radar reflectivity and is poorly constrained by observations e.g.. In this retrieval we use μ = 5, a value derived from both radar and distrometer studies . This simplifies the DSD to a two-parameter function of D0 and Nw. The uncertainty due to the assumption of fixed-μ DSD is estimated to be ±15 % of the rain rate , and is included in the uncertainty estimates of the retrieved quantities.

Our primary state variable is the rain rate, R=ρwπ6∫0∞N(D)D3v(D)dDkgm-2s-1, where ρw is the density of liquid water, and v(D) is the raindrop terminal velocity as a function of drop size from corrected for air density through the vertical profile. Hereafter we scale R by a factor of 3600 to express rain rate in units of mm h-1. For all retrievals a prior R of 0.1 mm h-1 is used. While a prior R is not strictly necessary, it is applied in combination with a large prior variance (σ(ln⁡R) = 4.0), such that the retrieved R is relatively insensitive to the prior unless the retrieval is poorly constrained by observations. We note that this value for the prior variance implies that before the measurements are taken we assume there is a 44 % chance of R lying between 0.01 and 1.0 mm h-1 and a 56 % chance that R is outside these limits.

The second state variable is Nw, so that one state variable is an integral over the DSD, and the second is a parameter of the DSD. Additional state variables increase the degrees of freedom of the retrieval, requiring more information from observational variables as constrains. Therefore we retrieve a single value of Nw for each profile with the physical interpretation of representing Nw as constant with height or as the vertically averaged value. The representation of Nw as constant with height is not expected to be borne out in cases where evaporation or collision–coalescence processes modify the drop number concentration through the vertical profile. We take as the prior Nw the number concentration intercept of the DSD, 8 × 106 m-4.

When few observational variables are available, a single-parameter retrieval of R can be made by assuming that Nw is constant and equal to its prior, thereby reducing the degrees of freedom so that R is a function of D0 alone. This is called the R-only retrieval and is similar to CloudSat rain rate retrievals in which Nw is assumed constant everywhere. When additional observational variables are available, such as the mean Doppler velocity, there may be sufficient information to also retrieve Nw; this is called the R - Nw retrieval.

We use the natural logarithms of R and Nw as the state variables with the effect that the values remain positive everywhere and that the algorithm converges in fewer iterations. While in moderate stratiform rain R is often close to invariant with height e.g., processes such as evaporation in the lower atmosphere and collision–coalescence in warm clouds will lead to significant variation with height in many contexts. R is therefore represented as the coefficients of a cubic spline basis function with n elements ; this has the effect of ensuring that the vertical profile of R is smoothly varying and continuous with height and also of reducing the number of terms in the state vector. Table  summarises the rain state variables, their prior values and uncertainties, and their physical representation in each vertical profile. For R-only retrievals the state vector x for a vertical profile is given by x=ln⁡R1⋯RnT, while for the R - Nw retrieval the state vector is x=ln⁡R1⋯RnNwT, where Nw is assumed constant with height in each profile.

We employ a simplified representation of the melting layer in stratiform precipitation by applying radar attenuation between the lowest pixel in each profile classified as ice and the highest pixel classified as rain, provided the two pixels are contiguous. The melting of graupel and hail, usually associated with convective precipitation, are not considered in this melting layer model. Following , it is assumed that the two-way attenuation of the melting layer A is proportional to the rain rate R at the first pixel just below the melting layer and the two-way path length Xm through the melting layer, such that A=kmXmR[dB], where the melting layer extinction coefficient km is 2.2 dB km-1 (mm h-1)-1 at 94 GHz and 0.04 dB km-1 (mm h-1)-1 at 9.6 GHz. The estimated attenuation through the melting layer is based on a Marshall–Palmer DSD for the rain below the melting layer and is not modified to match the retrieved DSD in the profile. The thickness of the melting layer and therefore the total attenuation may also depend on the local temperature profile: as sufficient information to retrieve the total melting layer attenuation may be available from the PIA and the attenuation inferred from the radar reflectivity gradient, we include the variable Xm in the retrieval to represent the effect of melting layer thickness on radar attenuation; however, in this study Xm is held constant with a value of 1.0 km, allowing us to capture the effect of this uncertainty on the retrieved variables and their errors without retrieving Xm.

For a given state vector we estimate the measurements made by each instrument by forward modelling the scattering behaviour between the sensor and each gate for the 94 and 9.6 GHz radars, accounting for the effects of atmospheric gases and hydrometeors.

The radar reflectivity factor of rain is a function of the sixth moment of the DSD, Z=∫0∞N(D)D6γf(D)dDmm6m-3, where γf is the Mie–Rayleigh backscatter ratio at the radar frequency f and is required for both 94 and 9.6 GHz radars to account for non-Rayleigh scattering. At 94 GHz the uncertainty of assuming raindrops are spherical Mie scatters is approximately 5 % in integrated backscatter for a gamma DSD with median drop size D0 = 1.5 mm when compared against estimates for oblate spheroids e.g. using the T-matrix method .

Scattering and attenuation effects are included in the radar forward model so that the forward-modelled estimate of the apparent radar reflectivity (Za) is directly comparable to observations. Attenuation due to atmospheric gases and the dielectric factor of water are calculated from atmospheric temperature and humidity profiles . Multiple scattering effects on radar and lidar backscatter can be estimated within CAPTIVATE using . Radar reflectivity enhancement due to multiple scattering is especially relevant to space-borne radar measurements at millimetre wavelengths , and the effects on Doppler radar measurements are expected to include both enhanced spectral broadening and modified mean Doppler velocity ; however, with the narrower beam of the airborne radar used in this study we can assume that multiple scattering effects are negligible .

Radar attenuation due to hydrometeors is quantified at each gate by the extinction coefficient k=π4∫0∞Q(D)N(D)D2dDm-1, where Q(D) is the extinction efficiency calculated from Mie theory . As for radar reflectivity, the uncertainty in extinction due to assuming spherical drops is less than 2 % for DSD with D0 of 1.5 mm. The gradient of extinction can be related to the gradient of apparent radar reflectivity and used to estimate the rain rate as suggested by . A second approach to quantifying attenuation due to hydrometeors is to measure the two-way path-integrated attenuation, PIA=210ln⁡10∫0∞kdz[dB], for each profile. PIA is derived from the radar reflectivity at the ocean surface and included as an observational measurement. Whereas in the gradient method is applied only at moderate to heavy rain rates where it can be assumed that the gradient of apparent radar reflectivity is dominated by attenuation, within the CAPTIVATE scheme both approaches are implemented simultaneously so that the gradient of R and k can be estimated from both the gradient of radar reflectivity and the PIA.

Finally the mean Doppler velocity is the reflectivity-weighted mean drop fall speed, v‾D=-∫0∞N(D)D6v(D)γf(D)dD∫0∞N(D)D6γf(D)dDms-1, where the terminal fall speed of drops v(D) is from the empirical formulation of scaled to account for air density changes with altitude and where positive velocities are toward the ground. The forward-modelled mean Doppler velocity is calculated assuming zero vertical air motion; therefore the difference between the forward-modelled and observed mean Doppler velocities will include a contribution from the vertical air motion, which is treated as an observational uncertainty.

The observed variables, their observational uncertainties, and their vertical representation are summarized in Table . The uncertainties in the observational variables include both the specified measurement errors for the instrument and the estimated uncertainties in the radar forward model. We have found that the weighting of errors between radar reflectivity and PIA is quite important for the retrieved rain rate and that if only instrument errors are included the retrieval is not sufficiently constrained by PIA. This is believed to be because attenuation affects all forward-modelled radar reflectivity measurements in the same way, leading to their having strong error correlations. Error correlations are not accounted for in the R matrix since they are profile dependent and difficult to estimate, which can lead to the radar reflectivity measurements being overweighted in the retrieval. To overcome this, we take the common approach e.g. of inflating the reflectivity errors (and in our case somewhat reducing the errors in PIA) to better balance the information coming from the reflectivity profile and from PIA.

In the moderate rain regime the ZvPIA retrieval estimates rain rates of 1.0–2.0 mm h-1 at the melting layer. In profiles with strong attenuation (PIA up to 20 dB), R is close to constant from the melting layer to the surface; conversely, in less attenuated profiles (with PIA around 10 dB) some evaporation is evident with R reducing to 0.1–1.0 mm h-1 at the surface (Fig. a). Estimates of Nw are consistently between 106 and 107 m-4 in this regime (Fig. c), close to the value, while D0 is around 1.0 mm at the melting layer and decreases somewhat toward the surface in profiles where evaporation is strong (Fig. d). Forward-modelled 94 GHz radar measurements agree with observations at 3 km (Fig. f–h), as expected since the retrieval minimizes differences between the observed and forward-modelled variables. The 9.6 GHz radar measurements forward-modelled from the retrieved state show generally good agreement with independent observations at 3 km a.sl. (Fig. i and j), although 9.6 GHz radar reflectivity is overestimated by as much as 3 dB in profiles with strong evaporation between 15:58 and 16:00 UTC, and mean Doppler velocity is underestimated in the profiles with the heaviest rain.

The averaged vertical profiles of the ZvPIA retrieval in moderate rain (Fig. ) show that the forward-modelled 94 GHz radar reflectivity is overestimated near the surface, while the largest error in 9.6 GHz is in the mean Doppler velocity in the lowest 2–3 km. We suggest that these errors in the forward-modelled variables through the vertical profile relate to the representation of Nw as constant with height such that the effects of evaporation on the DSD – a decrease in concentration of the smallest drops – is not resolved. The ZvPIA retrieval is broadly able to reproduce the 9.6 GHz radar reflectivity while slightly underestimating mean Doppler velocity.

The ZPIA and Zv retrievals illustrate the contributions of mean Doppler velocity and PIA to a ZvPIA retrieval and the ambiguities that arise in under-constrained retrievals. Both ZPIA and Zv retrievals are considerably more sensitive to the selection of priors and prior uncertainties than the ZvPIA retrieval. At 3 km a.s.l. (Fig. ), ZPIA estimates of R in the moderate rain regime are close to those of ZvPIA, but Nw and D0 differ significantly with ZPIA estimating a much higher concentration of smaller drops than ZvPIA. The forward-modelled mean Doppler velocity shows that this retrieval leads to large errors in drop fall speeds. Conversely, the Zv retrieval tends to underestimate rain rate in this regime by up to an order of magnitude, tending toward the prior R of 0.1 mm h-1 except in the strongly attenuated profiles between 15:57 and 15:58 UTC where Zv is, perhaps surprisingly, able to reproduce the observed PIA from the profiles of radar reflectivity and mean Doppler velocity. While D0 is well constrained by the mean Doppler velocity, without a constraint on PIA the forward-modelled observations confirm that the Zv retrieval tends to represent weakly attenuating profiles of rain; the forward-modelled 9.6 GHz variables show that this retrieval leads to a significantly underestimated radar reflectivity.

In the light rain regime, ZvPIA estimates R in the range 0.002–0.1 mm h-1 and Nw in the range 105–106 m-4. The lower rain rate corresponds to an observed 1.0 m s-1 decrease in 94 GHz mean Doppler velocity compared to the moderate rain regime; the retrieval resolves smaller drops in the light rain with D0 around 0.5 mm. The forward-modelled 9.6 GHz radar measurements from the ZvPIA retrieval are consistent with independent observations.

Zv retrieves R consistent with ZvPIA throughout the light rain regime, while ZPIA somewhat overestimates R in these profiles. PIA is negligible and provides little additional information in this regime; therefore the ZPIA retrieval represents a higher concentration of smaller drops as the retrieved R and Nw tend toward the priors. This sensitivity to the prior when observational information is limited was demonstrated in Sect. , and as in that synthetic case, the ZPIA retrieval here could be improved with a more appropriate prior. In contrast, with mean Doppler velocity as a constraint on drop size the DSD retrieved by Zv is very close to that of ZvPIA. The strong performance of Zv in light rain suggests potential for using Doppler radar for R - Nw retrievals of light rain over land.

The upper limit of the 94 GHz radar frequency for rain retrievals is reached in the heavy shower where PIA is saturated and no radar reflectivity or mean Doppler velocity is available below the melting layer. With limited observational constraints, both ZvPIA and ZPIA retrievals estimate R between 0.5 and 5.0 mm h-1; large uncertainties in R reflect the dearth of information available. The errors in forward-modelled 9.6 GHz radar measurements at 3 km suggest that the true rain rate lies on the upper end of this uncertainty range at around 10 mm h-1; this is confirmed by a retrieval assimilating both 94 and 9.6 GHz radar variables in Sect. . Without PIA information, the Zv retrieval interprets the deficit in radar reflectivity as a drop in rain rate and drop size, adding uncertainty to the retrieved quantities. The estimates of Nw vary over many orders of magnitude and are clearly unconstrained by observations in this regime, demonstrating that the R - Nw retrieval is not warranted without sufficient observational information. The PIA continues to provide a constraint on rain rate but becomes saturated once the radar is fully attenuated.

In this case of tropical stratiform rain the 94 GHz radar is fully attenuated by rain rates up to 10 mm h-1 falling from a melting layer at around 4.0 km a.s.l. In the midlatitudes, however, where melting layers are much shallower, successful R - Nw retrievals should be possible up to higher rain rates before the radar is fully attenuated.

A more comprehensive evaluation of the retrievals against independent 9.6 GHz radar measurements can be made using the joint frequencies of retrieved state variables (Fig. a–c) and forward-modelled 9.6 GHz radar measurements (Fig. d–f) for each retrieval. The major modes in the rain retrieval are evident in the distributions of R and Nw relative to the priors (dashed lines) and in the distribution of forward-modelled 9.6 GHz radar reflectivity and mean Doppler velocity compared against observations (black contours). In the 9.6 GHz radar variables the moderate rain regime exhibits radar reflectivity between 20 and 30 dB Z and mean Doppler velocity between 6 and 7 m s-1, while the light rain regime has radar reflectivity between 0 and 5 dB Z and a mean Doppler velocity of around 3 m s-1.

The ZPIA retrieval has a dominant mode corresponding to the moderate rain regime with R between 0.5 and 2.0 mm h-1 and a higher Nw with respect to the prior; without mean Doppler velocity this retrieval represents a relatively high concentration of small drops. The corresponding forward-modelled measurements shows that the small drop size leads to a significant underestimation of both mean Doppler velocity and radar reflectivity at 9.6 GHz.

Without PIA, Zv retrievals in the moderate rain regime tend toward weakly attenuated profiles with Nw less than 106 m-4, where R is close to the prior. This leads to underestimates of radar reflectivity by more than 10 dB; the mean Doppler velocity is reasonably well constrained but broadly underestimated by around 1 m s-1. A secondary mode with Nw close to the prior and R greater than 1 mm h-1 represents the strongly attenuated profiles of moderate rain in which Zv comes close to reproducing the observed PIA. Light rain profiles are represented with Nw ≈ 106 m-4, somewhat overestimating mean Doppler velocity.

ZvPIA resolves distinct modes for light and moderate rain regimes in the retrieved variables, and each mode corresponds well to the observed 9.6 GHz radar measurements: the moderate rain regime is represented with heavier rain than the Zv retrieval but with a lower concentration of smaller drops than the ZPIA retrieval; the light rain regime is similar to that of the Zv retrieval where the negligible PIA provides little additional information. Both rain regimes have Nw between 106 and 107 m-4, which is consistent with the average value of 2 × 106 m-4 for stratiform rain found by . The 9.6 GHz radar reflectivity is well represented across both rain regimes; however, the mean Doppler velocity shows that drop fall speed is slightly underestimated in moderate rain and overestimated in the light rain; this may be due to representing Nw as constant with height in each profile such that any variations in the DSD with height are expressed as changes in drop size rather than in drop number concentration.

We have retrieved R as a function of both D0 and Nw for a case of stratiform rain from melting ice, including rain rates from light rain as low as 10-3 mm h-1, to moderate rain with R up to 10 mm h-1. The retrieved Nw was around 106 m-4 throughout the case, which is consistent with expectations for average drop number concentrations in this context; the exception is in the heavy rain shower where the 94 GHz radar becomes fully attenuated, and insufficient information is available for R - Nw retrieval. The Zv retrieval, an analogue for Doppler radar retrievals over land, performed very well in light rain where PIA is close to zero but tended towards the priors in moderate rain. ZPIA retrievals without mean Doppler velocity tend to estimate R broadly accurately but retrieve DSDs with a high concentration of small drops, leading to errors with respect to the independent radar measurements; indeed, since the estimated Nw values were close to expectations in this context, a good non-Doppler retrieval of R could have been made by assuming that the value of Nw is equal to the prior.

The ZvPIA retrieval resolves a strong increase in rain rate from cloud top, where R is between 0.1 and 1.0 mm h-1, to the surface, where R increases to 1.0–10.0 mm h-1. Retrieved Nw is consistently around 1010 m-4 in the moderate rain regime, several orders of magnitude greater than estimated for rain from melting ice; accordingly, the drops are much smaller with D0 increasing from 0.1–0.3 mm at cloud top to 0.2–0.5 mm near the surface. At 1 km a.s.l. the 94 GHz radar measurements correspond very well to the forward-modelled variables. The 9.6 GHz radar reflectivity is also close to the forward model; however, while the forward-modelled mean Doppler velocity at 9.6 GHz also tracks well with observations, peaks associated with the heaviest precipitation features are not resolved.

The vertical structure of 94 and 9.6 GHz radar reflectivity is well represented in the ZvPIA retrieval over the moderate warm rain regime (Fig. ); however, mean Doppler velocity is underestimated by around 1 m s-1 in the lowest 1 km at both radar frequencies. The retrieval of constant Nw for each profile allows for a broadly satisfactory retrieval of the rain DSD with a good fit to observations, but the full vertical profiles show that some microphysical processes are not resolved: in warm rain we expect collision and coalescence to lead to both an increase in drop size and a decrease in drop number concentration toward the surface. It seems likely, as for the representation of evaporation in case 2, that while the retrieval of Nw allows for an improved retrieval of the DSD across a range of rain regimes, there are limits to the vertical variability in the DSD that can be resolved with a height-invariant Nw.

The ZPIA retrieval closely resembles ZvPIA; this includes matching estimates of D0 despite having no constraint on drop size from mean Doppler velocity. Zv retrieves similar D0 but underestimates Nw by as much as 2 orders of magnitude: the Zv-retrieved DSD has fewer drops and negligible PIA at 94 GHz, which corresponds to very large errors in forward-modelled 9.6 GHz radar reflectivity. Unlike the stratiform rain cases, here PIA is more important for an accurate retrieval than mean Doppler velocity: the mean Doppler velocity may be less sensitive to the changes in the terminal fall speed of small drops, while PIA in combination with radar reflectivity provides an effective constraint on the number concentration because only a DSD with many small drops satisfies the observed strong attenuation and low radar reflectivity.

In the light warm rain, ZvPIA estimates patchy precipitation features with R between 0.01 and 0.5 mm h-1 and D0 around 0.1–0.3 mm, similar to values at the tops of the deeper warm clouds but without significant drop growth toward the surface. The retrieved Nw in the lightest rain profiles is around 108–109 m-4 but returns to 1010 m-4 where heavier rain features are evident. The forward-modelled radar reflectivities are close to observations at 1 km, while the mean Doppler velocity again matches the lower range of measurements but not the peaks. ZPIA estimates R similar to ZvPIA in this regime, but without mean Doppler velocity in the lightest rain, fewer larger drops are retrieved with Nw tending toward the prior in some profiles. In contrast, Zv is very similar to ZvPIA in the lightest profiles with Nw consistently around 109 m-4.

In warm rain we have retrieved Nw several orders of magnitude greater than the Marshall–Palmer value with D0 in the range 0.1–0.5 mm in rain rates from very light drizzle up to 10 mm h-1 in the heaviest profiles. The contribution of PIA and mean Doppler velocity to R - Nw retrievals in warm rain differs from that in rain from melting ice: while Doppler is required to retrieve Nw when attenuation is low, it is possible to retrieve Nw without Doppler in strongly attenuated profiles of warm cloud where the combination of low radar reflectivity and high attenuation can only be due to a high concentration of small drops.