04 Oct 2021
04 Oct 2021
Novel assessment of numerical forecasting model relative humidity with satellite probabilistic estimates
 ^{1}Université ParisSaclay, UVSQ, CNRS, LATMOS/IPSL, 78280, Guyancourt, France
 ^{2}University of Oklahoma, Norman, Oklahoma, USA
 ^{3}NOAA/National Severe Storms Laboratory, Norman, Oklahoma
 ^{4}CNRM, Université de Toulouse, Météo France, CNRS, Toulouse, France
 ^{1}Université ParisSaclay, UVSQ, CNRS, LATMOS/IPSL, 78280, Guyancourt, France
 ^{2}University of Oklahoma, Norman, Oklahoma, USA
 ^{3}NOAA/National Severe Storms Laboratory, Norman, Oklahoma
 ^{4}CNRM, Université de Toulouse, Météo France, CNRS, Toulouse, France
Abstract. A novel method of comparison between an atmospheric model and satellite probabilistic estimates of relative humidity (RH) in the tropical atmosphere is presented. The method is developed to assess the MétéoFrance numerical weather forecasting model ARPEGE using probability density functions (PDF) of RH estimated from the SAPHIR microwave sounder. The satellite RH reference is derived by aggregating footprintscale probabilistic RH to match the spatial and temporal resolution of ARPEGE over the AprilMayJune 2018 period. The probabilistic comparison is discussed with respect to a classical deterministic comparison confronting each model RH value to the reference average and using a set confidence interval. The study first documents the significant spatial and temporal variability of the reference distribution spread and shape. It warrants the need for a finer assessment at the individual case level to characterise specific situations beyond the classical bulk comparison using determinist “best” reference estimates. The probabilistic comparison allows for a more contrasted assessment than the deterministic one. Specifically, it reveals cases where the ARPEGE simulated values falling within the deterministic confidence range actually correspond to extreme departures in the reference distribution.
 Preprint
(1748 KB) 
Supplement
(3871 KB)  BibTeX
 EndNote
Chloé Radice et al.
Status: final response (author comments only)

RC1: 'Comment on acp2021617', Anonymous Referee #1, 28 Oct 2021
GENERAL COMMENTS
================The paper describes a new method to compare satellite measurements against model
data leveraging inter quartile ranges derived from probability density functions.
The method is well introduced and explained and employed to compare SAPHIR
measurements against ARPEGE model data. It reveals significant and insightful differences
and discusses these in detail.The topic fits the journal.
The major short coming of this paper is the reference method chosen to be compared
against the newly developed method. As demonstrated by the paper itself, it is
not useful at all. The paper itself mentions the common use of second moments
for such comparisons but uses a blanket, unmotivated 15\% fixed error range itself.I recommend a major revision of the paper with a more commonly used reference method
e.g., one employing second moments (standard deviations).
MAJOR COMMENTS
==============This paper uses a very simple, so called "deterministic" method as reference.
The method assumes a blanket +15\% error range as acceptable, independent of the
actual level 2 data quality. This reference method is not properly motivated by the paper.
Also, often dominant errors in radiative transfer inverse problems are of a
multiplicative nature, which would affect high and low RH values very differently.
This fact alone makes a constant error range an unrealistic assumption.The analysis of the paper itself suggests that a smaller assumed range might be more suitable.
A very common method would be to use the standard deviation supplied by the data set (or at least compute it from
the available distributions, if not given directly); such a method has obvious short comings, particularly
for nonnegative quantities, but is an "industry standard".The paper must use a more reasonable reference method to compare against or demonstrate
that a blanket assumption of +15\% fixed offset error is a widely used method.
SPECIFIC COMMENTS
=================line 43
While it is true that the forward model introduces uncertainty into the comparison in
measurement space, almost all inversion schemes make use of a forward model (at least for
training a statistical model with obvious implications).
Due to the illposedness of the inversion, this implies that the uncertainty in geophysical
space is almost always larger than the uncertainty in measurement space, particularly as
the represntation in geophysical space might contain a large "nullspace" inaccessible
to the inversion (e.g. high frequency vertical oscillations in temperature to nadir sounders).
Thus large discrepancies in geophysical space might be very small in measurement space.
This is one of the reasons, why assimilation prefers assimilating radiances in contrast to
geophysical quantities (which are much easier to assimilate).The current text reads as if comparing in measurement space would be disadvantagous, while a
very strong case can be made for the opposite.A large disadvantage in comparing in measurement space is that
it is much more difficult to identify the reason for a disagreement in geophysical space
and thus the "faulty" model quantity.
line 103
Please provide an introduction to "beta probability density functions". The references in the
vicinity do not explain the term. A mathematical beta distribution has two free parameters, which seems
in principle feasible to derive for six layers from six BT measurements including error estimates.
I do not believe that most readers are familiar with the term such that it deserves a better introduction,
especially as it seem to lay the foundation for the latter IQR method.Also, one would derive by multivariate regression, under Gaussian assumptions, a maximum likelihood vector and
a covariance matrix detailing correlations in the data (optimal estimation). Typically the weighting functions of
the sounder are not sharp enough to neglect correlations...?Either way, please introduce the satellite level 2 product and its supplied diagnostics/error terms in sufficient detail.
line 131
Is the averaged PDF retained, which can be a rather arbitrary function (discretized in some fashion, I assume),
or is effectively only mean and sigma or the IQR computed? The example PDF look very Gaussianlike in all cases and
suggest such an interpretation. If the actual shapes are different, maybe some PDFs in the visualisation should
look more "wild".
line 163
The PDF suggests that a value of +15\% is too generous. Staying in a Gaussian framework this
looks like a 2sigma value, whereas the CDF based method with the CDF of 0.5 would correspond to
being in an interval of even less than +1sigma (being within one sigma has a probability of 68\%).The proposed method is sound, but the chosen example seems very biased. Even without using arbitrary
PDF functions, a Gaussian approximation and error analysis should be able to provide better results
than shown. Only if the PDF/CDF are nonGaussian, an improvement will be achieved.To that end, the authors should demonstrate the the difference to the (too) common Gaussian distribution
assumption is significant.
line 215
The interquartile range as central concept deserves at a onesentence explanation in addition to the backreference.
Using an IQR instead of the PDF looses a lot of information as it boils the arbitrary shape down to
two simple numbers, comparable to the Gaussian approach with mean/sigma. I do not expect large differences
unless strange, e.g., bimodal distributions, distribution appear.What is here the experience of the authors?
line 219
Again the 15\% uncertainty come up. The authors make a compelling argument against Gaussian models, but
picking a fixed 15\% uncertainty is much worse than a simple Gaussian modelbased uncertainty estimation would be.
With a deterministic uncertainty of, say, 30\%, the proposed method would compare even more favourably.Please provide a reference to the chosen value of 15\% being a reasonable error estimate for the level 2
product. Looking into some of the given references, I couldn't find it.Much better would be a comparison against a traditional Gaussian error analysis. The chosen confidence interval
can be compared against being within some factor times sigma of the derived value.
line 303
Showing the individual distributions of RH_mod and RH_obs would be interesting as well. The Talagrand diagram suggests,
as the authors note, that the model assume extreme values more often than the observation. The individual
distributions should show the same in a maybe more accessible/familiar manner.
Figure 8a
The colour scale of this figure hides a lot of detail as can be seen by the fact that nearly everything is gray. Another indication that the +15\%
assumption is not good. I bet a nonlinear colourscale blowing up the currently gray part would reveal a lot of interesting details.
line 426

Almost all level 2 satellite products from nadir or limb sounders offer a second moment (standard deviation) as diagnostics, many go beyond that (covariance matrices,
error terms from different sources). The analysis suggests that a 15\% error assumption is not reasonable for the current data set. Using a
proper second moment instead would certainly deliver more useful results.The employed method uses the more useful IQR, which is likely superior to a more simple first/second moment consideration.
This is, quite sadly, not demonstrated by the paper.
MINOR REMARKS
=============line 36
two way > two ways
line 92
km2 > km²
line 211
gaussian > Gaussian
line 394f
P>0.25 > P<0.25
Figure 9a

The colour scale should reflect the three regimes that have so far been used, ie. P< 0.25, 0.25<P<0.75 and 0.75 <P. AC2: 'Reply on RC1', Chloé Radice, 09 Jan 2022

RC2: 'Comment on acp2021617', Anonymous Referee #2, 21 Nov 2021
Novel assessment of numerical forecasting model relative humidity with satellite probabilistic estimates
Chloé Radice, Hélène Brogniez, PierreEmmanuel Kirstetter, and Philippe Chambon
1. General Comments
This paper presents a novel method for assessing humidity fields from numerical weather prediction models with estimates from the SAPHIR instrument. The probabilistic methodology used to estimate relative humidity from SAPHIR is exploited to provide a new approach for model assessment. The methodology also allows for a confidence interval to be placed on comparisons where classical ‘bulk’ comparisons.
This study demonstrates an innovation that yields more nuanced results for satellite and model intercomparisons. This is important for relative humidity, where uncertainties in satellite measurements can be as high as 10% RH for some instruments (especially heritage infrared sounders). Overall, I find that this study is of scientific value and recommend it for publication, after all the issues that I have highlighted are addressed.
2. Specific Comments
 Line 21: The final sentence in your abstract is illustrating a key point of your study but it is missing the “why” of its importance. Adding another sentence or editing this final one will make it more impactful.
 Line 42: Why use a reference for precipitation when talking about humidity? There are plenty of water vapour retrieval algorithm papers that perform an inversion between an atmospheric stat vector and observed radiances/brightness temperatures. Please update.
 Lines 4547: Averaging is not the only method used to get data on the same resolution. The discussion here does not include the use of averaging kernels, which are used to smooth model or in situ profiles relative to the vertical resolution of the satellite measurement. See “ Rodgers, C.D. and Connor, B.J., 2003. Intercomparison of remote sounding instruments. Journal of Geophysical Research: Atmospheres, 108(D3).”
 Lines 6567: It reads a bit strange when you talk about RH and then reference a precipitation paper for further discussion. If this is the only suitable reference there needs to be slightly more elaboration as to why. For instance, is the discussion point in the paper about representativeness but in the context of precipitation?
 Line 93: A figure here might illustrate this point better for the channels on SAPHIR. Not all readers may be familiar with MW remote sensing, especially the 183 GHz region where the +/ values relate to where on the wings of the 183 GHz feature SAPHIR is sampling. Alternatively, the sentence could be updated to reflect this point and why it is done.
 Lines 96108: Is the SAPHIR measurement noise (measurement uncertainty) used at all in the RH retrieval?
 Line 115: “RH fields range between 5 and +5 % (resp. 5 and 25%)” what do the values in brackets relate to?
 Line 120: Does the vertical averaging account for SAPHIR weighting functions? – in a similar way to which upper tropospheric humidity is calculated?
 Line 133: do you mean uncertainty in a metrological sense? If not you might want to change the word used. This is linked to the comment about lines 96108.
 Line 164: what is the uncertainty here? Source, magnitude? Or is it an error?
 Line 192: I don’t think you mentioned what you’re a priori error assumption is before this point, what is it? Do you get an a posteriori error? Do you calculate the error reduction?
 Figure 5: Did 12:00 UTC look different? Is there any correlation to convection?
3. Technical Comments
 Line 17: “.The study first …“ – change to “. This study first …”
 Line 18: “It warrants the need …”  this sounds like you are eluding to a future direction in a conclusion. Would something more like “We demonstrate the need …”
 Line 33: change “relies” to “rely”
 Line 72: “such a probabilistic approach.” – missing ‘a’
 Figure 1b: X axis label missing, also cannot see bars for values > 10, log scale might help here
 Line 137: “complementarities” – change to similarities
 Lines 232236: need a space between %RH, i.e. % RH. There is no need for a space between the value and the percent, e.g. 12% RH.
 Line 265: need a space between %RH, i.e. % RH
 Line 280: need a space between %RH, i.e. % RH
 Line 295: need a space between %RH, i.e. % RH
 Figure 6: need a space between %RH, i.e. % RH
 Figure 7: need a space between %RH, i.e. % RH
 Lines 343359: need a space between %RH, i.e. % RH
 Line 412: need a space between %RH, i.e. % RH
 Line 427: need a space between %RH, i.e. % RH
 Lines 439440 : need a space between %RH, i.e. % RH
 AC1: 'Reply on RC2', Chloé Radice, 09 Jan 2022
Chloé Radice et al.
Chloé Radice et al.
Viewed
HTML  XML  Total  Supplement  BibTeX  EndNote  

398  55  13  466  19  5  6 
 HTML: 398
 PDF: 55
 XML: 13
 Total: 466
 Supplement: 19
 BibTeX: 5
 EndNote: 6
Viewed (geographical distribution)
Country  #  Views  % 

Total:  0 
HTML:  0 
PDF:  0 
XML:  0 
 1