A new description of probability density distributions of polar mesospheric clouds
In this paper we present a new description of statistical probability density functions (pdfs) of polar mesospheric clouds (PMCs). The analysis is based on observations of maximum backscatter, ice mass density, ice particle radius, and number density of ice particles measured by the ALOMAR Rayleigh–Mie–Raman lidar for all PMC seasons from 2002 to 2016. From this data set we derive a new class of pdfs that describe the statistics of PMC events that is different from previous statistical methods using the approach of an exponential distribution commonly named the g distribution. The new analysis describes successfully the probability distributions of ALOMAR lidar data. It turns out that the former g-function description is a special case of our new approach. In general the new statistical function can be applied to many kinds of different PMC parameters, e.g., maximum backscatter, integrated backscatter, ice mass density, ice water content, ice particle radius, ice particle number density, or albedo measured by satellites. As a main advantage the new method allows us to connect different observational PMC distributions of lidar and satellite data, and also to compare with distributions from ice model studies. In particular, the statistical distributions of different ice parameters can be compared with each other on the basis of a common assessment that facilitates, for example, trend analysis of PMC.
First studies of probability distributions of polar mesospheric clouds (PMCs) were reported by Thomas (1995) using data from the UVS (ultraviolet spectrograph) instrument on board the Solar Mesosphere Explorer (SME) satellite and from the Solar Backscatter Ultraviolet (SBUV) instrument on the Nimbus-7 satellite over the period 1978–1986, measuring scattered limb albedo at 265 nm and nadir albedo at 273.5 nm, respectively. Thomas (1995) introduced empirical measures in the statistical analysis of PMC brightness distributions. He showed that the frequency distribution of PMC albedo derived from both SME and SBUV satellite data can be approximated by an (normalized) exponential probability function, see Fig. 3 in Thomas (1995). Secondly, the author also proposed to use cumulative frequency numbers (the so-called g function) of clouds, g(A), exceeding a certain albedo A in order to better represent the exponential populations. Examples of g distributions are plotted on a semilogarithmic scale in Fig. 4 in Thomas (1995), clearly indicating an approximately linear behavior of cumulative frequencies in a logarithmic format.
In the following years many observational PMC analyses of seasonal statistics have been published frequently using the g function, e.g., reports from Wind Imaging Interferometer (WINDII) and Polar Ozone and Aerosol Measurement II data (Shettle et al., 2002), SBUV data (Deland et al., 2003), Student Nitric Oxide Explorer (SNOE) data (Bailey et al., 2007), ice water content data derived from SBUV (DeLand and Thomas, 2015), or ALOMAR lidar data (Fiedler et al., 2017). Also model analyses have used the g function investigating trends and long-term changes in PMC parameters (Lübken et al., 2013; Berger and Lübken, 2015).
The g-function approach has been relatively successfully applied to many kinds of different PMC parameters as brightness, albedo, maximum backscatter ratio, integrated backscatter, ice water content, ice mass densities, ice particle size, or ice particle number density since frequency histograms of all these parameters have sometimes a nearly, at least piecewise, exponential shape. Furthermore, sometimes PMC data seem to fit almost perfectly to exponential distributions, particularly when using cumulative standardizations of data (Thomas, 1995). An example of a good exponential fit is the frequency distribution of ALOMAR backscatter data that are discussed in Sect. 3.1.1. On the other hand, in some statistical applications it is obvious that the exponential approach describes the data rather insufficiently, see examples of ice mass density, ice radius, and ice number density in Sect. 3.1.2. Therefore it is a desirable task to provide some more aspects on the theory of PMC statistics.
This paper makes an attempt to investigate in more detail the statistics of probability density functions (pdfs) of PMC climatology for various ice parameters. In the following we analyze a PMC data record of maximum backscatter, ice mass density, ice particle radius, and number density from the period 2002–2016 measured by the ALOMAR Rayleigh–Mie–Raman (RMR) lidar. From the analysis of these ALOMAR data, we derive a new class of pdfs of PMC distributions that, as we will show, modifies and improves the exponential (g-function) approach as introduced by Thomas (1995).
The data set obtained by the ground-based RMR lidar, located at the Arctic station ALOMAR (69∘ N, 16∘ E), consists of occurrence frequency, brightness, and altitude of PMC. The RMR lidar is in operation on a routine basis during the summer seasons (PMC season: 20 May to 20 August) since 1997. Since summer 2002 the lidar system has the general capability to run in a multiple wavelength (3-color) mode. We briefly summarize the 3-color lidar technique: laser pulses at three separated wavelengths (355, 532, 1064 nm) are emitted, scattered back by air molecules and ice particles in the atmosphere, and collected by telescopes. The received light is recorded by single photon counting detectors with an integration time of 15 min. After separation of the ice particle and molecular backscatter signal, we extract three vertical profiles of so-called backscatter coefficients, which are a measure of height-dependent brightness of the ice cloud. At the height of maximum backscatter (MBS) at 532 nm we calculate three MBS values. We assume that at the altitude of MBS, typically located near 83 km, the actual shape of the ice particle distribution can be described by a normal distribution. Then we derive from the three measured MBS values the characteristics of the normal distribution with mean ice radius, ice number density, and variance (Baumgarten et al., 2007). Finally, we also estimate from these ice parameters the actual ice mass density (IMD) at the MBS height. Such a Gaussian assumption has been widely used in PMC data processing of lidar and satellite data, e.g., ALOMAR lidar (Baumgarten et al., 2010) and AIM satellite with SOFIE/CIPS instruments (Hervig and Stevens, 2014; Bailey et al., 2015). Also microphysical model studies show strong evidence of Gaussian-distributed ice particles at the height of maximum brightness of PMC, e.g., Berger and von Zahn (2002) and Rapp and Thomas (2006).
In this paper we will analyze the climatology of all ice seasons from 2002 until 2016 merging all 15 seasons to one data record. Within this combined data set we then get a total number N of 8597 observations, which is sufficiently numerous in order to avoid excessive statistical irregularities in a frequency histogram of the data.
In general, the seasonal climatology of PMC events with measured ice parameters, such as integrated backscatter, maximum backscatter, column ice mass, albedo, or ice mass density, has been supposed to follow an exponential distribution that we name E(x) with ice parameter variable x. In the following we summarize the general characteristics of the exponential distribution that allows us to compute a numerical test for exponentially distributed data. The properties of the exponential probability distribution will be also compared with the characteristics of our new probability distribution approach introduced in Sect. 4.
The general form of the exponential distribution E(x) with scale parameter α>0 is defined as a pdf given by that fulfills the normalization condition of a pdf with . Thomas (1995) defined the g function g(x) as the cumulative probability Ecum with
Taking the logarithm of E yields a straight line . For a given class of values [x1;x2] the likeliness of this class is proportional to the area enclosed by the continuous probability distribution and is obtained by integrating E on the segment length (bin size) as .
A statistical analysis of ice parameters has to take into account the aspect of specific sensitivities of different instruments. For example the ALOMAR lidar is generally sensitive to a backscatter signal larger than a threshold about 2– m−1 sr−1 (Fiedler et al., 2017). When considering a threshold (xth) the exponential pdf E(x) is normalized according to with a scaling factor A=exp (αxth). We summarize the properties of the exponential distribution taking into account a threshold in Appendix A.
For a threshold of zero (xth=0) we get the regular exponential distribution E(x) that has the mean ; median ; mode η=0, which is the value that occurs most frequently in the data sample; variance ; and standard deviation . Note that the exponential distribution has the unique property that the mean μ and standard deviation σ are identical, see also Eqs. (A1) and (A4). In combination with the median (Eq. A2), these equations form a simple statistical constraint, namely
This allows us to test whether a given observational data sample shows good conformity with an exponential (g-function) distribution.
For a given data sample xi () assuming a threshold xi>xth we use the common estimates of mean and variance s2 (standard deviation s) with
In addition we also calculate the median and mode . Hence testing a data sample to be exponentially distributed means that mean, median, and standard deviation of the sample have to fulfill the following identity:
We will use this condition to analyze the ALOMAR data with respect to possible exponential (g-function) distributions.
3.1 Analysis of ALOMAR data on exponential distributions (g function)
3.1.1 Analysis of maximum backscatter data
We investigate the frequency distribution of MBS at 532 nm in units of 10−10 m−1 sr−1. We assume a threshold of 3 that corresponds to the instrumental sensitivity of the ALOMAR lidar. Then we sort the x = MBS data to a bin size of one per class starting from the threshold value and calculate a frequency histogram. Finally, we normalize the histogram so that the sum of all frequency classes equals one.
Figure 1a shows the frequency distribution of x = MBS data in a semilogarithm diagram. The first impression is that the data points are almost perfectly approximated by a linear regression besides some statistical noise. This indicates that an exponential function describes the distribution of data with a high accuracy. Figure 1b shows the distribution histogram in an original nonlogarithmic representation. We see that the exponential fit matches the data histogram with a high precision. The good quality of the fit is characterized by a small relative error of 6.5 % that is calculated as a sum of for x>3 with theoretical exponential frequencies Ej and normalized frequencies 𝒳j of data x per class j with a total of M classes. The high quality of the fit is also supported by the fact that theoretical mean, median, mode, and standard deviation (μ, ν, η, σ) using Eqs. (A1)–(A4) and estimates of mean, median, mode, and standard deviation (, , , s) from the data sample derived from Eq. (3) all coincide within their error bars. Now we perform the proposed exponential (g-function) test with , see Eq. (4), and insert the values from the data sample of mean (), median (), and standard deviation (). The error uncertainties have been estimated with bootstrap methods. We find that , , and . Hence the identity is fulfilled when allowing for uncertainties introduced by statistical errors. We conclude that lidar MBS data are very likely exponentially distributed and follow a g function.
3.1.2 Analysis of ice mass density, ice radius, and ice number density data
Now we investigate other ice parameters from the ALOMAR data set with respect to exponential distributions, namely the frequency distributions of IMD in units of mg m−3 (threshold 20, bin size of 2), ice radius r in units of nm (threshold 20, bin size of 1), and ice number density n in units of cm−3 (threshold 30, bin size of 10). We will show that these parameters do not follow an exponential distribution (g function). In Fig. 2a we plot the frequency distribution for y = IMD data in a semilogarithmic diagram. We show in the following that the data points have no dominant linear shape. There exist systematic deviations from data and the theoretical exponential fit. In comparison to the fit curve, data points are systematically smaller at y=20–40. Vice versa, data points substantially exceed fit values in the range y=40–90. Also, frequencies in all classes below the threshold are significantly smaller than a proposed exponential fit. Indeed, the frequency histogram in the nonlogarithmic frame shows these systematic deviations between data and exponential fit even more pronounced, see Fig. 2b. With a relative error of about 19 % the exponential curve fails to satisfactorily fit the data. Also, significant differences exist between fit and data parameters of mean, median, mode, and standard deviation (Fig. 2b). Finally, we apply the exponential (g-function) test for IMD data and get the following results: finding the mean (), median (), and standard deviation () directly calculated from the data sample, we get not equal to s=35.2 not equal to . Hence the condition of identity is not satisfied even allowing for uncertainties introduced by statistical errors again calculated from bootstrap methods. That is why we have to conclude that the lidar IMD data are very likely not exponentially distributed. When we investigate a possible exponential distribution for ice radius r and ice number density n data, see Fig. 2c–f, we even see larger discrepancies between data and exponential fits with, for example, relative errors of about 29 %, also indicating that both r and n are very likely not exponentially distributed. This is supported by the fact that the test of mean, median, and variance fails again and shows large inequalities.
We summarize that ice mass density, ice radius, and ice number density do not follow an exponential distribution in contrast to maximum backscatter. In the following section we will show that this is reasonable and is based on the fact that a functional link between MBS and the other data sets of IMD, r and n does miss a linear relationship.
3.2 Test of linearity between maximum backscatter and ice mass density, ice radius, ice number density data
Linearity between MBS and IMD, ice radius r and ice number density n data, is a necessary and sufficient condition that also shows that IMD, r and n data, samples are exponentially distributed, see next section. In the following we will test this constraint. Figure 3a shows a scatter plot in a logarithmic frame for simultaneously measured MBS and IMD data. In order to test a linear relationship between x = MBS and y = IMD we introduce a general fit function described by a power law condition as
with the two constants c (linear constant) and d (power constant). Only for d=1 we expect a perfect linear dependence between x and y. First of all, the logarithmic values of x and y do not yet have a high linear correlation (R=0.54), see Fig. 3a. Since the correlation coefficient R is unequal one, the two regression lines resulting from and differ from each other. This means that the best choice of a regression fit is determined by a straight line through the two regression points, which are defined by the means plus/minus standard deviations of logarithmic x and y data. Note that the positions of regression points also relate to the half-width of the angle that is spanned by the two regression lines y(x) and x(y). For our mean regression line we estimate d=0.873. The statistical error for d is with a confidence level of 95 %, which indicates a significant nonlinearity. Hence we conclude that the pdf describing the distribution of IMD data is very likely not an exact exponential function and its cumulative distribution does not follow precisely a g-function description because the criterion of “linearity” is violated. In Fig. 3b we show a second example for the correlation between MBS and ice radius n. Again the correlation is about R=0.55, but now the power constant is even much smaller with d=0.497, which is far away from unity. Finally, we investigated the linearity between MBS and ice number density n where we find a weak negative correlation of (not shown here). A best fit analysis yields a power value of , which again fails significantly the constraint of unity. Hence we conclude that ice radius and ice number density distributions should also not follow exponential (g-function) distributions.
In this section we will present the major part of the new statistical approach in order to describe frequency distributions of different PMC parameters.
There exists a general mathematical method (“integration by substitution”) that provides the opportunity to transform between pdfs with different statistical variables. This is done by the following procedure: assuming a given pdf P(x) with variable x, then the transformation from x to a new variable y(x) with a new pdf Q(y) is specified by
with x(y) being the inverse function of y(x). Here the absolute value of the derivative has to be calculated so that the new pdf Q is defined positively everywhere. In order to apply this approach one needs generally two requirements (1) Any transformation between the two pdfs, P and Q, needs an initial guess in one of the two pdfs, either P or Q. (2) An analytic formula of a forward and backward model must be available that describes the functional dependence between the two statistical ice variables x and y. In the following we discuss how we satisfy these two requirements.
We apply this method for two ice parameters, namely MBS with variable x and an unknown ice parameter named u (e.g., this unknown ice parameter might be ice particle radius). For condition (1), we use the hypothesis that the distribution of maximum backscatter data (MBS) is perfectly represented by an exponential pdf and its cumulative distribution is described by a g function according to Eq. (1). For condition (2), we assume a power form of a fit function used in Eq. (5) that also allows us to analytically calculate the inverse function. We discuss a suitable justification of this assumption in Sect. 6.2. Hence the forward model is u(x)=cxd and the backward model is . Then the new distribution 𝒰 for the arbitrary ice parameter u using Eq. (6) is given by
Equation (7) can be simplified to a more general form with
In a next step we introduce, in an arbitrary manner, a third ice parameter named z for which we assume again the same power law (Eq. 5) now valid between z and u as
Again we apply Eq. (6) and calculate the unknown pdf 𝒵(z):
At first glance the algebraic expression for 𝒵 looks particularly complex, but 𝒵 can be transformed to a general form with and as
Equation (9) represents our final result. The pdf 𝒵(z) describes the general form of the new statistical distribution. Note that the algebraic expressions of Eqs. (8) and (9) formally coincide. This means that any probability distribution of a new ice parameter that is connected to other ice parameters through our functional power law (Eq. 5) can be described by the general pdf given by Eq. (9). The constants a and b represent two free parameters in the 𝒵 distribution, which we name the scale parameter a and the shape parameter b. Obviously, the 𝒵 pdf is identical with an exponential pdf (or g function) in the limit b=1. This shows the close interconnection of the new 𝒵 pdf to the commonly used exponential (g-function) approach. We will show in the following that any distribution from so different ice parameters, such as maximum backscatter, ice mass density, ice radius, and number density of ice particles, can be described on a uniform basis with a high accuracy by 𝒵. Vice versa this indicates that these ice parameters are connected depending on each other by the uniform power law relation (Eq. 5), more details are discussed in Sect. 6.2.
5.1 General properties of the 𝒵 distribution
In this section we first show some general characteristics of the new 𝒵 distribution. From these properties we derive conditions and constraints that will allow to estimate the specific values of the two free constants in 𝒵, the scale parameter a and shape parameter b, for a given data sample.
First we show that 𝒵 is a correct pdf satisfying the normalization condition :
The definition range of is z≥0, a>0, and b≠0 with and
For a negative b the distribution 𝒵 is described by
The cumulative form of 𝒵 for b>0 is given by
For b<0 we have to choose the cumulative calculation in reverse order starting the integration at zero. Naming the reverse cumulative with index zero as we get
Only the cumulative descriptions from Eqs. (11a) and (11b) allow us, in principle, to roughly estimate the constants a and b for a given data sample using the double-logarithmic functional dependence, whereas the direct logarithm of 𝒵 (Eqs. 10a, 10b) offers no possibility to solve for a and b. However, the method calculating the double-logarithmic cumulative is not recommended. Several numerical tests showed that a stable estimation of a and b from noisy data applying this double-logarithmic approach is an almost impossible task. Instead, we propose two different methods that rely on much more powerful principles (see next Sect. 5.2). Additionally, we have to take care of a possible negative value of b that can be only identified using Eq. (11b). In fact such a case occurs in the analysis of ALOMAR data. In Sect. 5.3 we will give an example where only a negative slope parameter describes the distribution of number density of ice particles.
Generally, the 𝒵 distribution has the ability to characterize many different types of distributions, see Fig. 4. Especially the shape parameter b determines the shape of the 𝒵 distribution describing nonlinear exponential, exponential, right-skewed, left-skewed, or symmetric curves. For the pdf increase is nonlinear and exponentially accelerated to infinity as z approaches zero. For b=1 the pdf is exactly an exponential distribution having a positive finite value for z equal zero. For b>1, the function tends to zero as z approaches zero. When b is between 1 and 2, the function is right-skewed and rises to a peak quickly, then decreases for large z. When b has an approximate value between 3 and 4, the function becomes symmetric and bell-shaped like a normal distribution. Note that exact symmetry is given for a skewness equal to zero, which is true at z=3.60232. For b values larger than approximately 5, the function becomes again asymmetric changing the skewness to the left. For b<0, the function is skewed to the right and decreases steeply towards zero as z approaches zero. Note that 𝒵 is never negative and owns a local maximum described by the mode whenever b is negative or larger than 1. Finally we see that a double-logarithmic presentation of cumulative functions describes linear shapes with slope b, see Fig. 4j–l.
It is interesting to note that our new 𝒵 distribution is closely related to a more general Weibull distribution (Wilks, 1995). Nevertheless there is a difference concerning the shape parameter b, which in our case is not only defined for positive values but also for negative values. Such a case is disregarded by a classical 2-D Weibull distribution.
Now we shortly summarize the mathematical descriptions of median, mode, mean,
variance, and standard deviation parameters of 𝒵 for the case of a
zero threshold. The calculations are described in detail in Appendix B for
the general case of a nonzero threshold.
Variance and standard deviation:
The expressions of mean and variance use the gamma function . Notice that the gamma function is defined for all real values of t except t=0 and all negative integer values of t. Note also that median, mode, mean, variance, and standard deviation parameters of 𝒵 coincide with those of an exponential distribution in the limit as b equals 1.
5.2 Two computational methods to estimate the free parameters a and b of 𝒵 from a given data sample
In this section we present two numerical methods to calculate the scale parameter a and shape parameter b describing the new 𝒵 distribution. First of all, since any measurement depends on a specific instrumental sensitivity, we have to introduce a threshold that we name zth. The remaining data sample consists of N observations zi with zi>zth. Then we calculate the mean and standard deviation s of data zi using Eq. (3), and also the median value from data zi.
Method (1): we investigate the corresponding theoretical moments from 𝒵. In Appendix B we derive the theoretical mean μ (Eq. B5) and median ν (Eq. B3) for the 𝒵 distribution with a threshold constraint. Taking the estimates of mean and median from the sample as best proxies for the theoretical mean μ and median ν values of 𝒵, we get the following equations:
Note that the use of a threshold constraint involves the introduction of a scaling factor , which is present in Eq. (13b). Inserting the algebraic term of a (right side of Eq. 13a into the right side zero-equation Eq. 13b) and using the threshold value of zth yields an equation only for b, which has to be computed iteratively. Once a numerical value of b has been estimated with a sufficient accuracy, we insert this b value into the upper right equation to get the numerical value for a.
We note that in classical statistics the method of moments determines a and b from the mean and variance equations. In principle this approach should be possible here too, but in practise the algebraic structure of the variance equation is too complicated, see Eq. (B6) in Appendix B. This means that the variance equation, if at all, is only iteratively solvable, whereas the use of the median equation offers an analytical transformation to a. Generally, we recommend to apply the proposed method using the mean and median equations. This straight-forward method is easy to program and produces reliable estimates of parameters a and b.
Method (2): we also present a second method using a maximum likelihood approach, see Appendix C. The parameters are again calculated from two equations (Eq. C5) with
Interestingly, the left equation includes a term , which is the mean of the sample values weighted by power b, whereas the right equation includes the mean of logarithmic data and . This shows a similarity to the computation of regression points used in Fig. 3. We insert a into the right equation that yields a unique equation for b, which again can be solved iteratively. Once b is fixed, the left equation allows us to determine a. In the following we will test our lidar data samples with these two procedures and we will show that both methods produce almost identical results.
5.3 𝒵 distributions applied to ALOMAR data
Applications of the 𝒵 distribution to ALOMAR data of maximum backscatter (x), ice mass density (y), ice particle radius (r), and ice number density (n) are shown in Fig. 5. Note that thresholds have been computed from the regression functions (Eq. 5) described in Sect. 3.2 on the basis of m−1 sr−1, resulting in yth=22 mg m−3, rth=22.3 nm, and nth=662 cm−3. The values of scale parameter a and shape parameter b have been calculated with the method of mean and median equations (method 1). Then the theoretical curves of 𝒵 and theoretical values of mean, median, mode, and standard deviation have been calculated by inserting the values of a, b, and threshold zth into Eqs. (B1)–(B6). Obviously the pdf 𝒵 sometimes has no simple exponential shape, which is the case for ice mass density, ice radius, and ice number density. As we see in Fig. 5 all 𝒵-pdf curves (in blue) match the original data histograms with a high accuracy. The relative error is in a range of about 6 %–10 % except that ice number density has a relative error of 15 %. When we compare the mean, median, mode, and standard deviation derived from the theoretical distribution and corresponding estimates from data samples, we see a precise coincidence of mean and median values. Not surprising this is due to the fact that the parameters a and b have been computed by the mean and median method, which guarantees the preservation of mean and median values. Nevertheless standard deviation and mode also always show a good agreement within the error range. A closer look to the maximum backscatter distribution shows that MBS data are almost perfectly exponentially distributed with b=0.931, which is not too far away from b=1 for an exact exponential pdf. As we had already shown, see Sect. 3.1.1, MBS data are very likely exponentially distributed, now the 𝒵-distribution analysis confirms this result. Hence we conclude that the commonly used exponential (g-function) analysis might only be a reasonable statistical method in the case of analyzing MBS lidar data.
In contrast to MBS, the 𝒵 distribution of IMD shows a function that converges rapidly to zero for small IMD values. The distribution is described with b=1.355, which significantly deviates from b=1 for a precise exponential function. Note that the mode of the data sample at 40 mg m−3 differs from the theoretical mode of 23 mg m−3 because of a relatively high statistical noise in the data. But mean, median, and standard deviation values agree almost perfectly. Similar to IMD, the ice radius distribution indicates a significant nonexponential behavior with b=1.833. The distribution converges to zero as the radius approaches zero. The curve is skewed to the right and has a maximum at r=25.8 nm, which differs only slightly from the mode of the data sample at r=27.8 nm. Again mean, median, and standard deviation values agree almost perfectly.
The sample of ice number density shows a completely different behavior with a slope parameter that is negative with . The physical meaning is that the parameter ice number density is negatively correlated with all other ice parameters. For example, large ice numbers n correspond to small ice radii, IMD and MBS values. As a consequence this leads to a threshold of n in the reverse direction, that is from large values to small values defined by cm−3. One can see this feature in the right tail of 𝒵(n) plotted as a dashed curve, see Fig. 5d. The reverse behavior is also present for small values of n. Small values of n are measured for very bright PMC events with large MBS that have small occurrence rates. Therefore, the number of small ice particles has a relatively high uncertainty due to their low occurrence frequency, and it is this statistical error that produces some deviations from the fit curve to the data in the range of n=0–80 cm−3. We note that the numerical procedure computing the pair (a, b) from the method of mean/median (Eqs. 13a, 10b) has automatically detected the existence of a negative slope parameter b without any a priori information.
Now we repeat the analysis using method 2. Figure 6 summarizes the (a,b) values and statistical moments calculated from the method of maximum likelihood estimators. As can be seen the maximum likelihood approach computes almost identical results for all ice parameters. We have also added in Fig. 6a–c the histogram bars (in black) for all data being smaller than the threshold. Please keep in mind that the calculation of theoretical distribution curves is based exclusively on data larger than the threshold. Hence, decreasing or increasing a threshold will change the specific values of a and b. Figure 6d shows the ice number density distribution where we have added in the histogram (in black) all data being larger than the threshold. Again, also the maximum likelihood method has automatically detected the existence of a negative slope parameter b for the ice number density distribution.
6.1 Construction of artificial data
In the derivation of the 𝒵 distribution we used the assumption that all ice parameters of maximum backscatter x = MBS, ice mass density y = IMD, ice particle radius r, and ice number density n are connected with one another by the power law given in Eq. (5). In Sect. 5.3 we showed that the 𝒵 pdf describes with a high accuracy each distribution of these ice parameters, which, in turn, means that indeed there exists at least an approximative power law between ice parameters. We discuss a suitable justification of this power law relation in more detail in Sect. 6.2. In the following we will show that the use of a 𝒵 distribution allows us to construct artificial unknown data samples of various ice parameters that approximate true data to a high degree. We think that such an application is one of the most beneficial outcomes from the new 𝒵-distribution approach. We explain the numerical procedure by the help of a practical example.
We already showed a linear dependance in the logarithmic frame using linear regression (LR) for maximum backscatter and ice particle radius, see Fig. 3b. Hence we can compute artificial ice radius proxies , named as LR proxy of true data ri, as a function of MBS-data xi from the regression power law function (Eq. 5) with and with power law coefficients c=13.509 and d=0.497. Figure 7a shows a comparison between LR proxy and original ice radius data where we test the identity of the two data samples. The correlation coefficient is the same as shown in Fig. 3b with R=0.55. Mean and median values of proxy and original data are almost identical, and a regression analysis shows a perfect identity (c=1.000 and d=1.000). Now we calculate the frequency histogram of LR proxy ice radii, see Fig. 7b, and compare the histogram with the original 𝒵 distribution of ice radii already shown in Fig. 5c. We find that the LR proxy approximates the mean, median, mode, and standard deviation values of the original 𝒵 distribution with an relative error of 9.5 % comparable to the original error of 9.1 %. We conclude that a linear regression analysis of logarithmic data offers a good opportunity to approximate data, provided that a pair of data samples exists that allows for the calculation of power law coefficients c and d from regression methods.
Now the 𝒵-distribution approach offers a more general possibility to derive artificial data samples without any knowledge of correlation and regression coefficients. Indeed we will show that results from the 𝒵 approach are very close to results from a regression analysis. Again, our goal is to approximate ice radius data from a given maximum backscatter data sample. But now we suppose that no data of ice particle radii r exist, hence any correlation and regression analysis is not possible. First, we assume that a data sample of x = MBS of number N exists and also its 𝒵 distribution with ax=0.140 and bx=0.931 is well known, see Fig. 5a. Secondly, we assume that we know a priori the form of the 𝒵 distribution of ice radius r, e.g., with values of parameters ar and br from Fig. 5c (, br=1.833). Please keep in mind that such information about scale and shape parameters of the ice radius distribution could be also provided from independent satellite measurements that are capable of measuring ice particle radii, e.g., AIM-SOFIE.
Our new proxy method (𝒵 proxy) requires the following transformations. We first transform the xi values () into the z domain with followed by a second transformation with resulting in
Note that the derivation of c and d in Eq. (15) is based on the same mathematical steps when we developed the 𝒵 distribution from Eqs. (8) to (9). Inserting the ax, bx, ar, and br values into Eq. (15) determines the power law coefficients for 𝒵 proxy with c=12.994 and d=0.508. These values do not exactly coincide with c and d values obtained from the regression method, see above, but the identity test between 𝒵 proxies and true ice radii shows a very good coincidence, see Fig. 7c. Again, mean and median values of proxy and original data are practically identical, and a regression analysis shows an almost perfect identity (c=1.095 and d=0.980). Finally we calculate a frequency histogram of 𝒵 proxies, see Fig. 7d, and find a good agreement between proxies and true pdf. Mean, median, and standard deviations of 𝒵-proxy data correspond perfectly to original ice radius data, and the relative error has now even decreased to 9.2 %.
We summarize that we present a new method in order to construct artificial data samples provided 𝒵-descriptions of these data sets exist. By means of a consecutive arranging of ice parameters starting at a given data sample, this method allows us to construct any artificial data sample within . This method can also be applied to other data sets, e.g., ice parameter measurements from satellite observations. For example, a data sample of ice water content (IWC) obtained from satellite measurements might be analyzed in terms of a 𝒵 distribution estimating the scale and shape parameters a and b of the IWC distribution. This would allow us to establish a connection between satellite IWC data to lidar data samples through Eq. (15), hence the satellite IWC data could be transferred to lidar maximum backscatter, ice mass density, ice particle radius, and ice number density. Vice versa the knowledge of a satellite IWC 𝒵 distribution would allow us to transform lidar observations into IWC proxies and compare these with the original IWC observed by the satellite. We think that our proposed transformation method could be very helpful to connect different ice parameter data from different instruments, either from satellite observations or ground-based measurements. We also think that this new approach might be important in trend analysis of PMC.
In the next section we will discuss the power law assumption (Eq. 5) and the physical meaning of the shape parameter b, which might be introduced as a new trend variable in the analysis of PMC long-term changes.
6.2 Discussion of the power law assumption between PMC parameters
In this section we discuss some theoretical aspects of the power law dependence on ice parameters in order to validate the justification of Eq. (5). We use again the assumption as already discussed in Sect. 2 that at the altitude of maximum brightness (MBS) and ice mass density (IMD) there exists in the real atmospheric background an ice particle distribution that is perfectly Gaussian-distributed (𝒩i) as
We also assume that the geometric shapes of these ice particles are spheres with ice radii τ with mean radius ri and variance . Again index relates to the ith measurement in a given data sample of number N. 𝒩i is normalized to . When we assume an ice number density of ni particles per cubic centimeter we get the expression . Note that mean ice radii ri and ice number densities ni are elements of our lidar data climatology, which we have introduced in Sect. 2.
Furthermore, we assume from the analysis of lidar observations (three-color measurements) that the relation of mean radius and variance is according to σi=0.4 ri for ri<37.5 nm and σi=15 nm for ri≥37.5 nm (Baumgarten et al., 2010). This assumption has also been applied in the analysis of AIM/SOFIE-CIPS PMC satellite data (Lumpe et al., 2013; Hervig and Stevens, 2014). In order to simplify calculations we apply this relation σi=0.4 ri also for ri larger than 37.5 nm. We now investigate the question of which backscatter lidar and ice mass signals result from such an ice distribution.
We compute the mass of a spherical ice particle with radius τ as 4∕3πρiceτ3 with density of ice ρice=932 kg m−3. The backscatter signal from a single ice particle is calculated as aLτ5.8 with the lidar constant m2. Then the maximum backscatter xi and ice mass density yi are estimated by an integration of the radius distribution from zero to infinity as
assuming a constant number density n of ice particles. Only the integral of yi is analytically computable with a solution in which the error function defined by the integral erf is part of the solution:
The integral for xi includes the term τ5.8 that arises from Mie-scatter theory for light scattering of a wavelength of 532 nm (ALOMAR RMR lidar) at spheres in a range of radii with 1–100 nm. The exponential value of 5.8 approximates exact Mie-scatter calculations with a relative error less than 0.5 % in this radii range. Unfortunately, the integral can only be solved analytically if the exponent is an integer number as 5 or 6. Nevertheless, we are able to solve this integral by means of numerical methods with the specific exponent of 5.8. In a next step, we construct analytical approximations fx(ri) and fy(ri) for both integral solutions using a typical value of n=200 cm−3 with
The linear constants a1 and a2 with values a1=4.20 and a2=1.47 are optimal dimensionless parameters. fy approximates the analytical solution of yi with a relative error less than 0.7 % in the range nm], and less than 1.2 % in the range nm]. A precise solution of xi resulting from numerical methods of integration is approximated by fx with a relative error less than 0.3 % in the range nm], then the relative error increases linearly to a maximum error of 5 % at ri=70 nm. We find that the solutions fx and fy approximate the general power law condition p=cqd (Eq. 5) inside a small error range. Hence these analytical examples show that MBS is a function of ice radius proportional to ∼r5.8 (d=5.8), the same is also true for IMD (∼r3, d=3). It also follows that MBS and IMD are consequently connected through a power law condition with MBS ∼ IMD5.8∕3 ().
But the new form of the 𝒵-distribution technique opens up whole new perspectives for the validation of the analytical examples based on the ALOMAR lidar data samples. We transform the z distribution of IMD into the MBS domain using Eq. (15) with and that gives
We insert into Eq. (16) the values of ax=0.140, bx=0.931, , and by=1.355 from Fig. 5a and b and get c=0.023 and d=1.46. The power constant (d=1.46) derived from the shape parameters bx and by of the 𝒵-distribution analysis of real ALOMAR IBS and IMD data is significantly different from the power estimate (d=1.93) belonging to the analytical example that necessitates various assumptions, e.g., Gaussian-distributed ice particles at the height of maximum backscatter, constant ice particle number, or spherical shape of ice particles. Hence, we conclude that the determination of shape parameters b from a 𝒵-distribution analysis of observational data therefore provides a qualitative indication of the actual microphysical state that controls real ice formation processes. This leads to the idea that as a future task long-term changes in PMC formation might be characterized by potential long-term changes in b that indicate long-term changes in atmospheric background conditions and microphysical ice constraints of ice formation.
In this study we present a new method to describe statistical probability density functions (pdfs) for different ice parameters of PMC. We analyze a climatology of ice seasons from 2002 until 2016 as measured by the ALOMAR lidar. From this data set we derive ice cloud parameters of maximum backscatter, ice mass density, ice radius, and ice number density whose occurrence frequencies are investigated with respect to exponential distributions. We show that only maximum backscatter follows an exponential distribution, whereas ice mass density, ice radius, and ice number density frequencies fail to fit satisfactorily to an exponential distribution. The reason for these deviations from exponential behavior is based on the fact that these ice parameters are not linearly dependent on each other.
We introduce a new probability density distribution (𝒵 function, see Eq. 9) that instead assumes a general power law relation among ice parameters, see Eq. (5). The new 𝒵 distribution is described by two free constants with scale parameter a and the shape parameter b. We point out that the new distribution is closely related to a more general Weibull distribution. The new distribution has been applied to maximum backscatter, ice mass density, ice radius, and ice number density data from the ALOMAR data set. As a result all data distributions are described with a high accuracy by 𝒵. We discuss that the exponential distribution (g function) is a special case of the more general 𝒵 function with shape parameter b=1. We present two numerically stable methods (method of mean and median, method of maximum likeliness) that allow to derive the values of free constants a and b describing the actual 𝒵-function shape for a given data sample.
Perhaps the most important application of the new method is the possibility to construct unknown data sets for different ice parameters that approximate true data to a high degree. We show in Sect. 6.1 that a linear regression analysis in a logarithmic data frame offers a good opportunity to approximate data provided that a pair of data samples exists that allows for the calculation of power law coefficients c and d from regression methods. The 𝒵-distribution approach offers a more general possibility to derive artificial data samples without any knowledge of correlation and regression coefficients. This allows for the connection of different observational PMC distributions of lidar and satellite data, and also with distributions resulting from ice model studies. In particular, the statistical distributions of different measured ice parameters can be compared with each other on the basis of a common assessment that again should be helpful in combining trend analysis of PMC long-term time series from different observational data sets.
The ALOMAR lidar data are available at: ftp://ftp.iap-kborn.de/data-in-publications/BergerACP2019/ (Baumgarten, 2019).
When considering a threshold (xth) the exponential pdf E(x) is normalized according to with a scaling factor A=exp (αxth). It follows that the mean μ is then given by
This yields for the mean
The median ν denotes the boundary of separating the higher half from the lower half of the distribution with
The equation is solved for the median with
The mode is the value η at which AE(x) takes its maximum value
The variance σ2 in an exponential distribution considering a threshold is calculated with
Inserting simplifies the algebraic expression and shows that the variance σ2 (standard deviation σ) is independently from a given threshold:
In the following all quantities take into account a threshold zth. We introduce a scaling factor . Setting the threshold to zero means a scaling factor A=1 and gives the regular expressions for cumulative pdf, median, mode, mean, and variance, see Eqs. (12a)–(12d).
Probability density function :
Cumulative form of 𝒵cum:
Details of calculation:
Here we use the gamma function and the incomplete gamma function . Notice that the gamma function is defined for all real
values of a except a=0 and all negative integer values of a. The same applies to
Γ(a,x) with x≥0.
For a single observation, the likelihood function l of Z is calculated from Eq. (B1). Given a sample of N observations with threshold zth, the likelihood function is
Taking the logarithm of l yields the log-likelihood function
The derivative with respect to parameter a is
and for parameter b
Setting each of the derivatives equal to zero yields for a and b.
These are the maximum-likelihood estimators for scale parameter a and shape parameter b.
UB drafted the paper. All authors reviewed the paper and interpreted the data. JF and GB provided the lidar data from ALOMAR.
The authors declare that they have no conflict of interest.
This article is part of the special issue “Layered phenomena in the mesopause region (ACP/AMT inter-journal SI)”. It is a result of the LPMR workshop 2017 (LPMR-2017), Kühlungsborn, Germany, 18–22 September 2017.
We appreciate the financial support from the German BMBF for the ROMIC/TIMA project. We
thank Gary E. Thomas for very helpful and stimulating contributions, discussions, and
The publication of this article was funded by the
Open Access Fund of the Leibniz Association.
This paper was edited by Martin Dameris and reviewed by two anonymous referees.
Bailey, S. M., Merkel, A. W., Thomas G. E., and Rusch, D. W.: Hemispheric differences in Polar Mesospheric Cloud morphology observed by the Student Nitric Oxide Explorer, J. Atmos. Sol. Terr. Phys., 69, 1407–1418, https://doi.org/10.1016/j.jastp.2007.02.008, 2007. a
Bailey, S. M., Thomas, G. E., Hervig, M. E., Lumpe, J. D., Randall, C. E., Carstens, J. N., Thurairajah, B. T., Rusch, D. W., Russell III, J. M., and Gordley, L. L.: Comparing nadir and limb observations of polar mesospheric clouds: The effect of the assumed particle size distribution, J. Atmos. Sol. Terr. Phys., 127, 51–65, https://doi.org/10.1016/j.jastp.2015.02.007, 2015. a
Baumgarten, G.: ALOMAR RMR lidar NLC particle properties at peak of layer, available at: ftp://ftp.iap-kborn.de/data-in-publications/BergerACP2019/, last access: 1 April 2019. a
Baumgarten, G., Fiedler, J., and von Cossart, G.: The size of noctilucent cloud particles above ALOMAR (69∘ N): Optical modeling and method description, Adv. Space Res., 40, 772–784, https://doi.org/10.1016/j.asr.2007.01.018, 2007. a
Baumgarten, G. and Fiedler, J.: Vertical structure of particle properties and water content in noctilucent clouds, J. Geophys. Res. Lett., 35, L10811, https://doi.org/10.1029/2007GL033084, 2008.
Baumgarten, G., Fiedler, J., Lübken, F.-J., and von Cossart, G.: Particle properties and water content of noctilucent clouds and their interannual variation, J. Geophys. Res.-Atmos., 113, D06203, https://doi.org/10.1029/2007JD008884, 2008.
Baumgarten, G., Fiedler, J., and Rapp, M.: On microphysical processes of noctilucent clouds (NLC): observations and modeling of mean and width of the particle size-distribution, Atmos. Chem. Phys., 10, 6661–6668, https://doi.org/10.5194/acp-10-6661-2010, 2010. a, b
Berger, U. and von Zahn, U.: Icy particles in the summer mesopause region: Three-dimensional modeling of their environment and two-dimensional modeling of their transport, J. Geophys. Res., 107, 1366, https://doi.org/10.1029/2001JA000316, 2002. a
DeLand, M. T., Shettle, E. P., Thomas, G. E., and Olivero, J. J.: Solar backscattered ultraviolet (SBUV) observations of polar mesospheric clouds (PMCs) over two solar cycles, J. Geophys. Res., 108, 8445, https://doi.org/10.1029/2002JD002398, 2003. a
Fiedler, J., Baumgarten, G., Berger, U., and Lübken, F.-J.: long-term Variations of Noctilucent Clouds at ALOMAR, J. Atmos. Sol. Terr. Phys., 62, 79–89, https://doi.org/10.1016/j.jastp.2016.08.006, 2017. a, b
Hervig, M. E., Gordley, L. L., Stevens, M., Russell, J. M., Bailey, S., and Baumgarten, G.: Interpretation of SOFIE PMC measurements: cloud identification and derivation of mass density, particle shape, and particle size, J. Atmos. Sol. Terr. Phys., 71, 316–330, https://doi.org/10.1016/j.jastp.2008.07.009, 2009.
Lumpe, J. D., S. M. Bailey, J. N. Carstens, C. E. Randall, D. W. Rusch, G. E. Thomas, K. Nielsen, Jeppesen, C., McClintock, W. E., Merkel, A. W., Riesberg, L., Temple, B., Baumgarten, G., and Russell III, J. M.: Retrieval of polar mesospheric cloud properties from CIPS: algorithm description, error analysis and cloud detection sensitivity, J. Atmos. Sol. Terr. Phys., 104, 167–196, https://doi.org/10.1016/j.jastp.2013.06.007, 2013. a
Lübken, F.-J., Berger, U., Kiliani, J., Baumgarten, G., and Fiedler, J.: Solar variability and trend effects in mesospheric ice layers, chapter 18, in: Climate and Weather of the Sun-Earth-System (CAWSES): Highlights from a priority program, Springer Publ. Dordrecht, The Netherlands, 2013. a
Press, W. H., Teukolsky, S. A., Vetterling, W. T., and Flannery, B. P.: Numerical Recipes in Fortran: The Art of Scientific Computing, Second Edition, Cambridge University Press, ISBN 0-521-43064-X, p. 214, 1992.
Rapp, M. and Thomas, G. E.: Modeling the microphysics of mesospheric ice particles: Assessment of current capabilities and basic sensitivities, J. Atmos. Sol.-Terr. Phy., 68, 715–744, https://doi.org/10.1016/j.jastp.2005.10.015, 2006. a
Shettle, E. P., Thomas, G. E., Olivero, J. J., Evans, W. F. J., Debrestian, D. J., and Hardon, L.: Three satellite comparison of polar mesospheric clouds: Evidence for long-term change, J. Geophys. Res., 107, 4134, https://doi.org/10.1029/2001JD000668, 2002. a
Thomas, G. E.: Climatology of polar mesospheric clouds: Interannual variability and implications for long-term trends, in: The Upper Mesosphere and Lower Thermosphere: A Review of Experiment and Theory, edited by: Johnson, R. M. and Killeen, T. L., Geophys. Monogr. Ser., 87, 185–200, 1995. a, b, c, d, e, f, g
von Cossart, G., Fiedler, J., and von Zahn, U.: Size distributions of NLC particles as determined from 3-color observations of NLC by ground-based lidar, J. Geophys. Res., 26, 1513–1516, 1999.
Wilks, D. S.: Statistical Methods in the Atmospheric Sciences, International Geophysics Series, edited by: Dmowska, R. and Holton, J. R., Academic Press, Inc., San Diego, California, 59, 1–467, ISBN 0-12-751965-3, 1995. a