Dependence of predictability of precipitation in the northwestern Mediterranean coastal region on the strength of synoptic control

The weather-regime-dependent predictability of precipitation in the convection-permitting kilometric-scale AROME-EPS is examined for the entire HyMeX-SOP1 employing the convective adjustment timescale. This diagnostic quantifies variations in synoptic forcing on precipitation and is associated with different precipitation characteristics, forecast skill and predictability. During strong synoptic control, which dominates the weather on 80 % of the days in the 2-month period, the domain-integrated precipitation predictability assessed with the normalized ensemble standard deviation is above average, the wet bias is smaller and the forecast quality is generally better. In contrast, the pure spatial forecast quality of the most intense precipitation in the afternoon, as quantified with its 95th percentile, is superior during weakly forced synoptic regimes. The study also considers a prominent heavy-precipitation event that occurred during the NAWDEX field campaign in the same region, and the predictability during this event is compared with the events that occurred during HyMeX. It is shown that the unconditional evaluation of precipitation widely parallels the strongly forced weather type evaluation and obscures forecast model characteristics typical for weak control.

1 Introduction

The Mediterranean region is affected by intense precipitation events every year, particularly during the autumn months. Very high rain amounts and ensuing flash floods can cause widespread damage. Accurate prediction of these precipitation events is crucial to take precautions, warn the public and mitigate potential consequences. The HyMeX (Hydrological Cycle in the Mediterranean Experiment) field campaign was designed to advance the knowledge of Mediterranean heavy-precipitation and flash-flooding events, to improve numerical models, and to examine the representation and predictability of high-impact weather events (Ducrocq et al., 2014, and references therein).

Precipitation represents a very important yet challenging forecast variable due to the involvement of many atmospheric variables and the role of inherently highly non-linear processes in its formation. Forecasting precipitation with numerical weather prediction models requires, among others, a sufficiently fine model resolution to explicitly represent important processes like deep convection and a precise description of the microphysical processes leading to precipitation, but also an ensemble approach to quantify the forecast uncertainty.

In the last decade ensemble prediction at the convection-permitting kilometric scale has become a standard technique for weather forecasting and provides an important tool to forecast uncertainty. However, the intermittency and spatiotemporal variability of precipitation on the kilometric scale render the assessment of accuracy even more difficult. Next to the probabilistic approach, an evaluation of high-resolution ensemble forecasts of precipitation calls for spatial measures to assess forecast quality.

The predictability of weather in general, and precipitation in particular, is weather regime dependent (Anthes, 1986; Bauer et al., 2015; Yano et al., 2018). The prediction of precipitation is influenced by the synoptic-scale environment and local processes and instabilities. Previous results suggest that there is higher forecast quality and above-average predictability, that is lower uncertainty during strong synoptic control. The convective adjustment timescale offers an objective measure to classify weather regimes into strong and weak synoptic control on precipitation (e.g. Keil et al., 2014; Surcel et al., 2017). For instance, Schwartz and Sobash (2019) apply this diagnostic and conclude that forecast quality is related to forcing strength, with higher accuracy in more strongly forced regimes over the conterminous United States.

In the present study we aim to systematically identify different predictability regimes of precipitation in southeast France and northwest Italy during autumn 2012, for which the HyMeX campaign offers an unprecedented transnational observational dataset to validate convective-scale ensemble prediction systems (Ducrocq et al., 2014). This period extends from 5 September to 5 November 2012 of which 59 d experienced noteworthy precipitation and includes numerous well-studied intensive observation periods (IOPs) of high-impact weather situations. Here, forecasts of the kilometric-scale AROME-EPS system are evaluated for the first time with neighbourhood methods to examine the spatial distribution of precipitation and to infer on different predictability levels conditional upon the weather regime of the day.

Previously Bouttier et al. (2016) evaluated the AROME-EPS system over the full HyMeX-SOP1 period and identified strengths and weaknesses. The impact of initial conditions and model surface perturbations shows a significant effect on the ensemble performance. Using a variety of conventional scores like rms errors and spread–skill relation complemented with the probabilistic measures rank and relative operating characteristics (ROC) diagrams applied on near-surface variables, they found specifically for precipitation an almost negligible impact of direct surface perturbations and a lack of spread. The present study extends this comprehensive work, focuses on ensemble precipitation forecasts over the contiguous 2-month period and adds the aspect of weather-regime-dependent predictability.

Nuissier et al. (2016) presented a probabilistic evaluation of two convection-permitting ensemble prediction systems (EPSs) for the full HyMeX-SOP1 and document a slightly better performance of AROME-EPS forecasts in terms of discriminating behaviour and reliability of 6-hourly rainfall. However, results depend on the choice of the verification domain since small areas suffer from sampling issues and the occurrence of precipitation events. The examination of the HyMeX “golden case” (IOP16a on 26 October 2012, Ducrocq et al., 2014) reveals slightly different predictability levels in two different subdomains in which precipitation is crucially governed by the location and deepening of a surface low-pressure system over the Mediterranean Sea that is controlling the southerly moist low-level flow. Earlier, Hally et al. (2014) investigated the sensitivity of precipitation forecasts in an experimental convective-scale ensemble based on the Meso-NH model to diverse initial and boundary conditions and microphysical uncertainties for two IOPs (IOP6 and IOP7a in southeast France). Since both cases developed under strong synoptic forcing, the impact of atmospheric conditions on the spatiotemporal distribution of precipitation outweighs that of microphysical and surface perturbations. It is suggested that the specific influence of surface conditions is larger for weakly forced events. Recently, Fourrié et al. (2019) revisited the HyMeX-SOP1 period and demonstrate improved rainfall forecasts with a second reanalysis using 24 % more additional data in the AROME system. The superior performance is specifically illustrated employing the conventional score frequency bias and equitable threat score on 24 h rainfall accumulations for an intense precipitation event that occurred over Spain and southern France on 29 September 2012 (IOP8).

Beyond HyMeX the downstream impact of synoptic systems on the predictability of high-impact weather in the Mediterranean has been one of the science goals of the NAWDEX (North Atlantic Waveguide and Downstream Impact Experiment) campaign in autumn 2016 (Schäfler et al., 2018). One of the NAWDEX highlights represents IOP9 on 13 October 2016 when the 24 h accumulated precipitation in southeast France reached 250 mm ahead of the cyclone Sanchez. This prominent case is included in the present study to be compared with the 2-month HyMeX-SOP1.

This article focuses on precipitation predictability examining temporally highly resolved forecasts (3-hourly) of the AROME-EPS and relates differences in ensemble spread and forecast skill to broader environmental characteristics for the entire HyMeX-SOP1 period. The remainder of the paper consists of a methods section, followed by a classification of the 2-month period into weather regimes, an illustration of three prominent cases, the verification using classical grid-point-based quality measures, probabilistic metrics and a spatial score to allow for location tolerance, and finally conclusions.

2 Model, data and metrics

2.1 The convective-scale ensemble

The AROME-EPS used in this present study is based on the AROME forecasting system largely described in Seity et al. (2011) and in Brousseau et al. (2016). It is based on adiabatic, non-hydrostatic equations from the limited-area ALADIN (Aire Limitée Adaptation dynamique Développement InterNational) model (Bénard, 2004; Bubnova et al., 1995). A horizontal resolution of 2.5 km and 60 (HyMeX ensemble) or 90 (NAWDEX ensemble) vertical levels are used in this study. AROME shares the same physical parameterizations as the research model Meso-NH (Lac et al., 2018), including a bulk one-moment microphysics scheme following Caniaux et al. (1994), which represents six water species (water vapour, cloud water, rain water, primary ice, graupel and snow). The representation of the turbulence in the planetary boundary layer is based on a prognostic turbulent kinetic energy (TKE) equation combined with a diagnostic mixing length (Bougeault and Lacarrère, 1989). The TKE scheme used in AROME was developed by Cuxart et al. (2000), and the scheme is derived from the full set of equations for second-order moments. At 2.5 km resolution, deep convection is assumed to be explicitly resolved by the model’s dynamics. However shallow convection requires a parameterization of subgrid effects for which the Pergaud et al. (2009) scheme is used. It is a mass flux scheme based on the eddy diffusivity mass flux (EDMF) scheme (Soares et al., 2004) that parameterizes dry thermals and shallow cumuli.

The AROME-EPS ensemble setup is the following. (i) The ensemble comprises 12 members. For the HyMeX period, ensemble simulations start at 00:00 UTC (up to 36 h forecast range), whereas ensemble runs are initialized at 21:00 UTC (up to 45 h forecast range) for the NAWDEX case. (ii) In the ensemble simulations, AROME is driven by the global short-range ARPEGE-EPS (Descamps et al., 2014), hereafter called PEARP. Firstly, a subset of 12 members of the PEARP is selected according to the Nuissier et al. (2012) technique. The PEARP 35-member ensemble forecasts are classified by a complete-linkage clustering technique (Molteni et al., 2001). (iii) The initial conditions are provided by adding downscaled forecast perturbations of the selected PEARP members to the AROME operational analysis (Raynaud and Bouttier, 2017). (iv) Atmospheric model errors are represented through the so-called SPPT scheme (stochastic perturbation of physics tendencies) described in Bouttier et al. (2012), which simulates the effect of random errors due to physical parameterizations. (v) Finally, random perturbations are added to various parameters of the Surface Externalisée (SURFEX) surface scheme, including for instance sea-surface temperature, soil moisture and temperature perturbations (Bouttier et al., 2016).

Figure 1 Map illustrating the northwestern Mediterranean domain (black box) and geographical landmarks used in the text.

2.2 Domain and observational data

The investigation domain extends across 300 km × 800 km and encompasses southeastern France and northwestern Italy including the coastal regions of Côte d'Azur and the Riviera as well as adjacent mountainous regions of the Massif Central and the Alpes-Maritimes (Fig. 1). This region, which is herein called the northwestern Mediterranean, is prone to heavy precipitation generated by a wide variety of flow conditions including synoptic systems characteristic of Rossby wave breaking at the eastern end of the North Atlantic storm track, modulated by orography and thermal contrasts of the Mediterranean basin as well as calm, conditionally unstable situations requiring trigger mechanisms to generate rainfall (e.g. Ducrocq et al., 2014; Nuissier et al., 2016, and references therein). The location and size of the investigation domain are carefully chosen and represent a compromise between being large enough to have numerous precipitation events giving good statistics, but small enough to comprise a specific and unambiguous meteorological situation in combination with the good coverage of rainfall observations in the northwestern Mediterranean. If the domain is too large, strongly differing meteorological systems may be contained and the results obtained using area averages may be blurred and not representative. We believe that the chosen domain encompassing 300 km × 800 km represents a good compromise being below the scale of the Rossby radius of deformation (𝒪(1000 km)). The domain size conforms with the recommendation of Wernli et al. (2009) to use areas smaller than 500 km × 500 km to compute an unequivocal spatial forecast quality value representative of a certain meteorological situation. We examined sub-domains with a longitudinal extent of 500 km as well and got similar results. However, from a hydrological point of view the best choice might be a more intuitive domain like a river catchment.

Figure 2 Time series of forecasted area-averaged ensemble mean precipitation (blue), convective adjustment timescale (τ_c, green), CAPE (orange) and normalized spread of precipitation (S_n, grey) in the northwestern Mediterranean coastal domain for the entire SOP1 period. Thick lines represent temporally smoothed data using a 24-hourly moving average, whereas thin lines indicate the raw 3-hourly data. Note that the y coordinate has a logarithmic scale, CAPE is divided by a factor of 10, S_n is multiplied by a factor of 10 and precipitation is labelled on the right-hand side to increase readability.

As observational data we use 3-hourly rain-gauge observations retrieved from the HyMeX database and hourly rain-gauge observations for the NAWDEX case that are accumulated 3-hourly. The rain-gauge observations are spatially interpolated to the model grid using a linear barycentric interpolation to perform the spatial evaluation. We are aware that pointwise rain-gauge measurements can miss rain when the rain coverage is low and has local spikes, typically for weakly forced convective situations.

2.3 Metrics and measures

Generally, an ensemble of forecasts provides a range of possible scenarios allowing for the estimation of forecast uncertainty. Large deviations of individual ensemble members point towards a large forecast uncertainty and a below-average predictability. One method to estimate the predictability of precipitation is the computation of the normalized standard deviation S_n (e.g. Hohenegger et al., 2006; Nuissier et al., 2016). In the present study S_n is calculated at any grid point where the 3-hourly ensemble mean precipitation rate exceeds 1 mm (3 h)⁻¹ and is subsequently area averaged. Larger S_n values indicate higher ensemble dispersion, larger forecast uncertainty and lower predictability.

A hierarchy of measures is applied to conduct the weather-regime-dependent verification of precipitation forecasts during HyMeX-SOP1. Following the grid-point-based spread (SD) and root-mean-square error (RMSE), we present the two probabilistic measures relative operating characteristics (ROC) and reliability diagram (Wilks, 2011; Jolliffe and Stephenson, 2012) complemented with the widely used fraction skill score (FSS; Roberts and Lean, 2008) to account for spatial tolerance.

Figure 3 Scatterplot of domain-averaged, daily maximum convective adjustment timescale and daily mean normalized standard deviation of precipitation for the entire SOP1. The prominent cases discussed in Sect. 4 are highlighted: the red circle represents HyMeX IOP16a, the blue circle HyMeX 11 September 2012 and the white circle the NAWDEX case. The size of the symbols indicates the daily precipitation accumulation, with the grey circle in the bottom right corner displaying a daily domain-integrated rainfall accumulation of 10 mm for comparison.

2.4 The convective adjustment timescale

The convective adjustment timescale τ_c constitutes an objective measure to classify weather situations by taking the ratio of convective instability (measured by convective available potential energy, CAPE) and its removal (expressed by the precipitation rate; Done et al., 2006; Keil et al., 2014). During synoptically forced weather, precipitation balances the production of instability generated by, e.g. large-scale ascent, the atmosphere is in equilibrium and the value of the convective adjustment timescale is small. In contrast, when the synoptic forcing is weak, local processes like solar insolation, or the interaction with orographically generated convergence lines, are necessary to overcome a barrier (e.g. convective inhibition) and release convection. During this non-equilibrium regime large CAPE values can build up before convection is triggered and the convective adjustment timescale becomes larger and more comparable to the synoptic timescale.

Figure 4 Aggregated mean daily precipitation divided into strong (a, c) and weak (b, d) forcing conditions. Displayed are ensemble mean (a, b) and interpolated rain-gauge observations (c, d) of 24 h precipitation for the 2-month period in autumn 2012.

Figure 5 Aggregated diurnal cycle of the ensemble mean convective adjustment timescale (a) and 3-hourly precipitation (b) averaged over the full SOP1 period (black) and over weakly (blue) and strongly forced weather regimes (red) across the northwestern Mediterranean domain.

The area-averaged τ_c value can be used to categorically determine the weather regime of the day (e.g. Keil et al., 2019). Firstly, Gaussian smoothed forecast fields of 3-hourly precipitation rates and most unstable CAPE are taken to calculate τ_c at any grid point exceeding 3 mm (3 h)⁻¹ in individual members given that a minimum areal coverage of this precipitation rate is reached. The half-width size of the Gaussian kernel amounts to 20 grid points, and a threshold value for precipitation needs to be used, since dry grid points preclude any τ_c computation. Both values are set to be consistent with previous work (e.g. Keil and Craig, 2011; Kühnlein et al., 2014; Keil et al., 2014, 2019). Secondly, the ensemble mean of individual τ_c values at any grid point is computed, and thirdly, the domain average of this ensemble mean is taken. Finally, if the maximum domain averaged ensemble mean τ_c exceeds a threshold criterion at least once a day, that day is classified to be weakly forced.

In the present study a threshold value of 3 h is chosen to account for the smaller τ_c values occurring in the autumn season of HyMeX-SOP1 (see Fig. 3). Previously a threshold value of 6 h was used to separate mid-latitude precipitation regimes depending on dynamic control in the summer season (Kühnlein et al., 2014; Keil et al., 2019), whereas a value of 3 h was applied in a maritime environment to characterize convective regimes over the British Isles (Flack et al., 2016). The choice of these values concurs with results of Zimmer et al. (2011), who argue that the τ_c diagnostic results in a continuous distribution and conclude that a value somewhere between 3 and 12 h clearly distinguishes between different regimes. It turns out that a threshold of 3 h substantially reduces the sampling error, giving a distribution of 48 strongly and 11 weakly forced days during HyMeX-SOP1.

Figure 6 Time series of ensemble mean area-averaged precipitation, its normalized standard deviation S_n, ensemble mean CAPE and the convective adjustment timescale τ_c for the prominent cases (a) HyMeX IOP16a, (b) HyMeX 11 September 2012 and (c) NAWDEX 13 October 2016. Precipitation time series of the individual ensemble members highlights intra-ensemble variability. Additionally area-averaged rain-gauge observations are shown. Note the different scaling of τ_c on 11 September 2012.

3 Classification based on the strength of synoptic control

At first glance the time series spanning the entire 2-month period shows the variability of weather on daily timescales in autumn 2012 (Fig. 2). The precipitation curve highlights some of the golden cases observed during HyMeX-SOP1 (e.g. IOP6 on 24 September, IOP7a on 26 September, IOP8 on 29 September, IOP16a on 26 October) with peaks exceeding 2 mm (3 h)⁻¹ in domain-integrated precipitation. The time series of convective instability (CAPE) exhibits large variations, too. High values of spatially averaged CAPE exceeding 100 J/kg mostly concur with the occurrence of strong precipitation events (e.g. IOPs 6, 7a, 8 and 16a) pointing towards their predominantly convective character, whereas sometimes maxima do not coincide (e.g. beginning of September). During these episodes convective instability is created by, for instance, solar insolation but cannot be removed by precipitation because inhibiting factors like capping inversions atop the boundary layer prevent convection initiation. The rank correlation of CAPE and 3-hourly precipitation (and its normalized standard deviation S_n) amounts to 0.44 (and 0.28, respectively) and confirms the limited predictive power of CAPE alone.

Here the convective adjustment timescale τ_c provides a more suitable measure than CAPE to distinguish and to classify weather situations with different synoptic control. Using a categorical threshold of 3 h for the daily maximum area-averaged convective adjustment timescale results in 48 strongly and 11 weakly forced days in the northwestern Mediterranean domain during HyMeX-SOP1. Many of the weakly forced cases occur in the first week of the SOP1 (8 to 11 September, Fig. 2). After mid-October there are no weakly forced cases anymore, suggesting the influence of the seasonal cycle, as decreased solar insolation limits diurnally driven precipitation. However it is the interplay between the creation of convective instability and its removal by precipitation (both variables make τ_c) that shows the overall decrease in autumn that is strongly modulated by the occurrence of mid-latitude weather systems. During SOP1 τ_c exceeds the threshold value ultimately on 13 October, while area-averaged CAPE maxima exceeding 100 J/kg still occur in late October (e.g. on 26 October, IOP16a). The comparison of the time series of τ_c and of the normalized standard deviation S_n in Fig. 2 gives a first indication of a connection between both, that is between the weather regime and the forecast uncertainty. Large values of τ_c indicating weakly forced weather conditions correspond with above-average values of S_n, suggesting below-average precipitation predictability.

This relationship and clear dependence of the convective adjustment timescale τ_c and the normalized standard deviation S_n of precipitation is further illustrated in Fig. 3. Large values of τ_c correspond with large S_n of precipitation being a sign of below-average predictability. The strongest precipitation events (e.g. IOP16a and the NAWDEX case) occur predominantly at low τ_c values when the normalized ensemble spread of precipitation (S_n) is small, too. Thus, in a domain integrated sense, synoptically forced situations cause higher precipitation accumulations with lower forecast uncertainty. The rank correlation between τ_c and S_n is 0.6, providing statistical evidence that τ_c can be reasonably used as a predictor to classify weather regimes with inherently different precipitation predictability. Moreover, the scatterplot shows that the majority of τ_c values amounts to less than 3 h, and a comparison with Fig. 4 in Keil et al. (2019) confirms that the chosen threshold value represents a sensible classification criterion in this specific region at that time of the year.

Figure 7 Illustration of 6-hourly precipitation for IOP16a valid 26 October 18:00 UTC: (a) ensemble mean precipitation forecast (initialized 26 October 00:00 UTC) and observations (filled circles); (b) normalized ensemble spread (grey) and rainfall observations (filled circles) for reference. Panels (c, d, e, f) show 6-hourly precipitation of selected individual ensemble members, all overlaid with 10 (thin) and 20 mm (thick line) ensemble mean precipitation contours.

The different dynamical control shows its fingerprint in the mean spatial distribution of daily rainfall, too (Fig. 4). Apparently, there is more precipitation during strongly forced weather situations than during weakly forced ones. The regions receiving more than 5 mm/24 h during synoptic control at the southeastern foothills of the Massif Central, the western slopes of the Alpes-Maritimes and the Mediterranean coast east of Marseille agree with observations (Fig. 4a, c; see Fig. 1 for geographical landmarks). In contrast, a spottier distribution of daily rainfall concurrent with an overestimation of precipitation totals becomes evident during weak synoptic control (Fig. 4b, d, and later in Fig. 8).

The aggregated diurnal evolution during weak synoptic control shows the characteristic behaviour with a pronounced diurnal cycle of τ_c peaking in the early afternoon shortly before the maximum precipitation rates occur in the late afternoon (Fig. 5). Conversely, during strong forcing conditions there is almost no diurnal pattern in τ_c, and the precipitation rates are higher and show a weaker amplitude with maxima in the early evening. The unconditional average of τ_c and precipitation is fairly similar to the strongly forced weather type because these flow conditions are prevalent during the HyMeX period, thus dominating the diurnal evolution.

4 Three prominent and representative cases

In this section three characteristic cases are presented to highlight the different nature of individual events in detail and to identify hypotheses to be proven in the subsequent systematic evaluation using a hierarchy of measures. The prominent cases comprise HyMeX IOP16a (26 October 2012), a typical weakly forced situation during HyMeX (11 September 2012) and the NAWDEX Sanchez case (13 October 2016). The daily time series of precipitation, its normalized standard deviation S_n, the ensemble mean CAPE and τ_c clearly depict their different character (Fig. 6). For IOP16a and the NAWDEX case, τ_c is always considerably smaller than the threshold criterion (and even less than 1 h, Fig. 6a, c), indicating strong synoptic control (as for IOPs 6, 7a; not shown) in agreement with Hally et al. (2014) and Nuissier et al. (2016), whereas the temporal evolution of precipitation, CAPE and τ_c on 11 September 2012 shows the characteristic behaviour of weakly forced weather situations (Fig. 6b), that is high τ_c values preceding the strongest rainfall (compare to Keil et al., 2014). On this weakly forced day the normalized spread S_n is evidently higher and precipitation amounts are overestimated.

Figure 8 Same as Fig. 7, but for 11 September 18:00 UTC (initialized 11 September 00:00 UTC).

4.1 Strongly forced case on 26 October 2012 (IOP16a)

HyMeX IOP16a represents a case of deep convection that developed over the western Mediterranean Sea and affected the coastal regions of France and Italy. The synoptic situation was characterized by a deep upper-level low centred over the Iberian Peninsula moving slowly eastward and fuelling slowly propagating mesoscale convective systems in the northwestern Mediterranean. IOP16a represents a golden case enabling us to address the predictability of a high-impact weather event (Ducrocq et al., 2014; Nuissier et al., 2016).

Figure 7 depicts the spatial distributions of 6 h ensemble mean precipitation, its intra-ensemble variability and four individual ensemble members (all 12:00 to 18:00 UTC). Three hotspots of precipitation are forecast on 26 October. The spatially largest is located across the Massif Central, where the ensemble mean exceeds 10 mm/6 h, the intra-ensemble variability (represented by S_n) is small and all members indicate widespread precipitation (see Fig. 1 for geographical landmarks). At the southeastern foothills of the Massif Central the ensemble overestimates the 6 h precipitation (exceeding 20 mm/6 h) and there is a considerable ensemble spread. There, forecasts of single members diverge (Fig. 7c–f) and show a displacement of the heaviest precipitation (e.g. shifted eastward in member 7, very intense and southward in member 8). In the Rhône valley, an area where the ensemble mean indicates more than 5 mm, the intra-ensemble variability is large and individual members fail to predict any precipitation (e.g. member 12).

A second hotspot of strong precipitation occurs in the Var region where maximum rainfall accumulations are observed (larger than 50 mm/6 h). There, larger values of S_n point towards a higher intra-ensemble variability that becomes apparent when looking at the rainfall sums of individual members (e.g. Fig. 7e, f). A third heavy-precipitation region is forecast close to Genoa in Italy. Observations indicate a considerable overprediction in this region with filled circles depicting rain-gauge observations being clearly recognizable (Fig. 7a). However, the hidden circles across large regions of the Massif Central and the Alpes-Maritimes point towards the overall good performance of the ensemble mean forecast of precipitation. Interestingly precipitation amounts under strong synoptic control are strongly modulated by orography with the highest intra-ensemble variability in the flat regions of the Rhône valley and south of the Massif Central.

4.2 Weakly forced case on 11 September 2012

Weather on 11 September represents a characteristic case of a weakly forced situation during HyMeX. Before noon single convective cells are triggered with very different intensity and location in the individual members (not shown), resulting in very small area-averaged rainfall accumulations (Fig. 6). Subsequently, convection intensifies, leading to more than 50 mm/6 h in individual members at different locations (Fig. 8c, d, f). The differences in terms of exact location of the heaviest precipitation result in maximum ensemble mean values of less than 20 mm (Fig. 8a). Overall, precipitation is strongest across mountainous regions (a more or less distinct precipitation band extends from southwest to northeast across the Massif Central in all members) with correctly forecast dry conditions south of 44^∘ N in the Var region. Whereas the accumulated precipitation distribution in members 1 and 3 resembles the ensemble mean pattern, other members exhibit big deviations: member 4 forecasts hardly any rainfall, while member 9 forecasts a lot of precipitation in the western part of the domain only. Large S_n values west of 6^∘ E demonstrate this considerable intra-ensemble variability. Overall, the comparison with rainfall observations suggests a notable overestimation (Fig. 8a) and a clear connection to orography during weak control.

The weather situation on 11 September is characteristic for the first week of the HyMeX-SOP1 period (see Fig. 2) when solar insolation in early autumn is still strong enough to generate convective instability by surface heating, resulting in large CAPE and large τ_c values indicating a need for local triggering mechanisms to overcome convective inhibition and to form precipitation (see Fig. 6).

4.3 Heavy precipitation on 13 October 2016 during NAWDEX

The detailed examination of individual heavy-precipitation events in the western Mediterranean region is complemented with one of the most prominent cases in that region that developed downstream of the cyclone Sanchez during the NAWDEX field campaign in autumn 2016 (Schäfler et al., 2018). Figure 9 shows a good match of forecast and observed 24 h precipitation peaking in the southern foothills of the Massif Central with more than 200 mm. This event is clearly classified as a strongly forced regime with an area-averaged maximum τ_c of less than 30 min (see Figs. 3 and 6).

Figure 9 Observed (circles) and forecast (initialized 12 October 21:00 UTC) ensemble mean accumulated daily precipitation until 14 October 2016 06:00 UTC.

Figure 10 Same as Fig. 7, forecast for 13 October 2016 18:00 UTC (initialized 12 October 21:00 UTC).

However, and unexpectedly for a case under strong control, individual ensemble members show surprisingly large spatial variability that becomes evident when inspecting the time window of heavy rainfall in the afternoon (between 12:00 and 18:00 UTC, Fig. 10). Focussing on the region of the heaviest precipitation (20 mm/6 h in ensemble mean) at the southern foothills of the Massif Central, all members exhibit strong precipitation rates individually, whereas in other areas there are large discrepancies (e.g. members 5 and 6, Fig. 10d, e). Within the heaviest precipitation region all members agree well, resulting in small S_n values. Above-average precipitation variability occurs northward (across the Massif Central) and across the Mediterranean Sea.

In summary, the three selected cases indicate that the heaviest precipitation is co-located with orography during both regimes, that the spatial predictability of precipitation can considerably vary from case to case even within one forcing type and that the precipitation intensity is overestimated during weak control.

5 Systematic verification conditional on the strength of synoptic control

Figure 11 Aggregated time series of the spread (standard deviation) and skill (RMSE of the ensemble mean) averaged over the full SOP1 period (black) and over weakly (blue) and strongly forced weather regimes (red).

Finally, we examine the question of how precipitation forecasts during the different weather regimes compare with observations using a hierarchy of measures applied on the full 2-month period. Firstly, we show the mean diurnal evolution of the grid-point-based root-mean-square error (RMSE) of ensemble mean 3-hourly precipitation forecasts and rain-gauge observations conditionally averaged on both weather regimes.

During strong control the RMSE exhibits less diurnal variation than during weak control when a typical diurnal cycle is recognizable, attaining the highest error during the convectively most active period in the afternoon between 12:00 and 18:00 UTC (Fig. 11). The magnitude of the error reaches values up to 1.2 mm (3 h)⁻¹ in the northwestern Mediterranean, which is roughly 50 % less than found by Bouttier et al. (2016) looking at large parts of western Europe. Given that rainfall rates during weak forcing amount to only about 60 % of the rates during strong forcing (Fig. 5b), the relative error is higher in the weak regime. Likewise, the ensemble spread shows a diurnal cycle and is highest during the convectively most active period in the afternoon under weak synoptic control. Since 80 % of the days during HyMeX-SOP1 are classified as strongly forced weather regime, it is not surprising that the regime-independent curve follows the strongly forced curve closely, thus obscuring the forecast model characteristic during weak control.

Secondly, the regime-dependent probabilistic performance of the ensemble is investigated using the ROC and reliability diagrams for 3-hourly (Fig. 12a, b) and daily accumulations (Fig. 12c, d). Both probabilistic scores highlight the superior performance during strongly forced weather regimes. The larger distance of the ROC curve points from the diagonal (resulting in larger concavity) during strong control indicates greater event discrimination when 3-hourly (and daily) precipitation accumulations are averaged over the entire SOP1. In this weather regime the AROME-EPS forecasts are generally more reliable, in particular when averaged over 24 h (Fig. 12d). The calibration functions in the reliability diagrams show that the forecast probabilities are consistently too large relative to the conditional observed relative frequencies. This is an indication of overforecasting equivalent to an overconfidence of the ensemble that is strongest during weak synoptic control for short (3-hourly) time windows (Fig. 12b). The overconfidence could come from a wet bias or a tendency to have too much spatial agreement. Moreover, the flatness of the calibration function for this weather regime reveals a poor resolution. Observed relative frequencies depend only slightly on the forecast probabilities and always amount to less than 20 % for all forecast probabilities of moderate 3 mm (3 h)⁻¹ precipitation rates. Relaxing the temporal exactness and extending the window to daily accumulations improve the reliability, in particular during strong control (Fig. 12d).

Figure 12 Relative operating characteristics (ROC) (a, c) and reliability diagram (b, d) for 3 mm (3 h)⁻¹ precipitation rates (a, b) and daily amounts of 10 mm (24 h)⁻¹ (c, d) conditional on weather regime aggregated over the entire HyMeX-SOP1.

Finally, the fraction skill score (Roberts and Lean, 2008; Faggian et al., 2014) is employed to address the double-penalty problem inherent in convective-scale precipitation forecasts. To compute the FSS we threshold each member first, then take the ensemble mean of the binary probabilities and then apply the FSS. In Fig. 13 the ensemble mean FSS is shown as a function of neighbourhood size for absolute rainfall rates (0.3 mm (3 h)⁻¹, a threshold frequently used to separate rain versus no-rain areas, and 10 mm (24 h)⁻¹ accumulation) split into weather regimes aggregated for the 2-month period. During strong forcing the spatial forecast quality of the low rainfall threshold is superior for all neighbourhood sizes (Fig. 13a). The skill increases when relaxing the grid point proximity and comparing larger neighbourhoods, as expected. The fairly large box sizes and the extension of the whiskers demonstrate the large variability of forecast quality during the 2-month period. The upper quartile of the boxplot is touching upon a FSS value of 0.5 at neighbourhood sizes of 150 km during strong forcing. A FSS value of roughly 0.5 is also known as the believable scale (FSS $=\mathrm{0.5}+f_{\mathrm{0}}/\mathrm{2}$, where f₀ is the observed precipitation coverage; see Dey et al., 2014), a scale at which forecasts are deemed reasonably skilful and useful (Roberts and Lean, 2008). Thus, 25 % of the time (3-hourly intervals on 48 strongly forced days, i.e. for 96 time windows within SOP1) the forecasts are skilful at a scale of 𝒪(100 km), which is of the same order as found in previous studies (Clark et al., 2010; Mittermaier et al., 2011; Schwartz and Sobash, 2019; Bachmann et al., 2020), based on FSS and other neighbourhood methods. Useful precipitation forecasts are hardly encountered during weak forcing using absolute rainfall rates.

Relaxing the temporal exactness of 3-hourly accumulations towards daily sums confirms previous results. Using a fixed precipitation amount of 10 mm per day reveals that the mean FSS during strong control always lies higher than during weak control; that is on average the spatial forecast accuracy is higher during strongly forced weather situations, as expected (Fig. 13b). The discrepancy between the mean and the median of FSSs suggests that the high threshold of 10 mm (24 h)⁻¹ represents rare events with different intermittency characteristics in forecast and observation, leading to a skewed distribution.

The inspection of the spatial forecast accuracy of the prominent cases again highlights the large day-to-day variability. Both strongly forced prominent events (IOP16a and NAWDEX case) exhibit a very good spatial forecast quality, with the FSS reaching values larger than 0.8 for window sizes larger than 50 km tantamount with the highest whiskers (Fig. 13). The NAWDEX case (occurring in 2016) even shows FSS values higher than the highest whiskers found during HyMeX. The excellent forecast performance is mainly caused by the low precipitation threshold (0.3 mm (3 h)⁻¹) and the widespread precipitation occurring on both days. Large parts of the domain receive such precipitation rates and the FSS attains high values. The prominent weakly forced case indicates an average forecast performance (FSS of 11 September matches the mean value of this regime) for low rainfall rates separating essentially rain and no-rain regions.

Figure 13 Fraction skill score (FSS) of ensemble mean 3-hourly precipitation vs. interpolated rain-gauge observations as a function of neighbourhood size conditional on weather regime: (a) 0.3 mm (3 h)⁻¹ and (b) 10 mm (24 h)⁻¹, all averaged for the entire HyMeX-SOP1. The horizontal line (dashed line) within the boxes indicates the median (mean). The boxes and whiskers are slightly displaced at the discrete window sizes (weak to the left, strong to the right) to increase readability. Additionally, the FSS of the prominent cases is depicted.

Figure 14 (a) Aggregated diurnal cycle of 3-hourly precipitation values corresponding to the 95th percentile, both stratified in weakly (blue) and strongly forced weather regimes (red) and averaged over the full SOP1 period. (b) FSS of 95th percentile of 3-hourly precipitation as a function of neighbourhood size conditional on weather regime averaged for the entire HyMeX-SOP1 between 12:00 and 18:00 UTC.

However, taking into account a varying model bias during different weather regimes changes the picture. The pure forecast location accuracy neglecting a model bias can be estimated by using percentiles of forecast and observed precipitation amounts. Whereas the  amounts corresponding to the 95th percentile of forecast and observed precipitation agree well during strong forcing (at least until 18:00 UTC), there is a considerable overprediction during weak forcing (Fig. 14a). This overforecasting is strongest during the convective most active period in the afternoon between 12:00 and 18:00 UTC. Taking this bias into account by using precipitation percentiles results in a superior spatial forecast quality during weakly forced regimes (Fig. 14b). Thus forecasting the pure location of the heaviest precipitation in the afternoon (expressed by the 95th percentile) is better during comparably quiescent synoptic-scale atmospheric conditions. This is at first sight an unexpected and surprising result. Given the favourable meteorological ingredients for generating deep convection at this specific geographical region in the autumn season (Grazzini et al., 2020), we hypothesize that well-represented steady land surface structures (like orography, particularly) in kilometric-scale models provide sufficient trigger mechanisms to initiate convection and serve as a permanent source of precipitation predictability during weak control. The structuring effect of mountains on the location of precipitation has previously been shown in idealized and real-world ensemble simulations of summertime convection in central Europe (Bachmann et al., 2019, 2020). In contrast, forecasting the pure location of the heaviest precipitation with high temporal exactness at forecast horizons of 12 to 24 h is challenging during transient synoptic-scale weather systems typical during strong control.

6 Conclusions

This study extends prior work documenting the performance of AROME-EPS during HyMeX-SOP1 (Bouttier et al., 2016; Nuissier et al., 2016) by the weather-regime-dependent aspect of precipitation predictability with a special focus on the spatial forecast quality. The convective adjustment timescale τ_c is used to categorically classify every single day within the 2-month period in autumn 2012 into one specific weather type depending on the strength of the synoptic control. From a physical perspective, it is sensible to use variations in forcing (i.e. τ_c), rather than CAPE, as being associated with variations in precipitation characteristics and forecast skill (Schwartz and Sobash, 2019).

Altogether, the ever-changing meteorological situations in the northwestern Mediterranean coastal region are stratified into 48 strongly and 11 weakly forced days during HyMeX-SOP1. All weakly forced (i.e. locally triggered) precipitation events occur before mid-October with a sequence of weakly forced days in the beginning of September. This distribution follows the seasonal cycle and reflects the climatological study of heavy-precipitation events in northern Italy showing that weakly forced events occur from mid-May to the end of October with the highest frequency from mid-August to mid-September (Grazzini et al., 2020). Key HyMeX IOPs are classified as strongly forced weather types in agreement with literature (Hally et al., 2014; Ducrocq et al., 2014; Nuissier et al., 2016). Likewise, the prominent heavy-precipitation event that occurred during NAWDEX is clearly identified as strongly forced (Schäfler et al., 2018).

A clear connection between the weather regime and (i) the mean diurnal evolution of precipitation, (ii) the mean spatial distribution of daily rainfall, (iii) the precipitation predictability, (iv) the precipitation bias, (v) the probabilistic forecast quality and (vi) the spatial forecast quality is found. During strong synoptic control, which dominates the weather on 80 % of the days during HyMeX-SOP1, the domain-integrated precipitation predictability assessed with the normalized ensemble standard deviation S_n is above average, the wet bias is smaller and the forecast quality is generally better. Conversely, there is a pronounced diurnal cycle of area-averaged precipitation and a considerable intra-ensemble variability in terms of placement of precipitation (i.e. large S_n) during weakly forced weather types consistent with previous results (e.g. Keil et al., 2019; Schwartz and Sobash, 2019; Bachmann et al., 2020). Disregarding the wet bias during weak control by focussing on the 95th percentile of precipitation shows a superior pure spatial predictability of most intense precipitation in the afternoon during weak control. We hypothesize that a reasonable representation of steady land surface structures (e.g. orography, coast line) in kilometric-scale numerical models provides trigger mechanisms to initiate convection during weak control and serves as a source of location predictability for precipitation, given favourable atmospheric conditions in this special geographical region. The important role of orography in precipitation in this region in this season is in agreement with the climatological study of Grazzini et al. (2020), who found that convective precipitation is largely influenced by orography during the frontal uplift with embedded equilibrium deep convection (herein strong control) as well as non-equilibrium convection (herein weak control). One reason for the apparent overprediction of precipitation during weak control can partly be accounted for by the pointwise character of rain-gauge measurements that sample the spatially highly heterogeneous and intermittent nature of locally triggered convective precipitation insufficiently. This discrepancy calls for remotely sensed spatial rainfall measurements of high quality that were not available in the present study.

It is shown that the unconditional evaluation of precipitation widely parallels the strongly forced weather type evaluation and might obscure forecast model characteristics typical for weak control. Such a separation of statistics according to local weather conditions might prove useful to improve physical parameterizations that depend on the weather condition, as, for instance, Bouttier et al. (2012) suggested for the correlation lengths in the stochastic SPPT scheme, to identify a regime-dependent impact of certain surface perturbations (Baur et al., 2018) or to enhance nowcasting capabilities (Kober et al., 2014).