The data sets on which this study's analyses are based originate from two extensive measurement campaigns performed during the HD(CP)2 Observational Prototype Experiment (HOPE) using a set of autonomous silicon photodiode pyranometers. These instruments measure the downwelling shortwave radiation at the Earth's surface in the range between 0.3 and 1.1 µm. Although this wavelength band does not span the entire solar irradiance spectrum, it corresponds well with the relevant bandwidths of typical semiconductor materials used for photovoltaic applications . In fact, the pyranometers themselves may essentially be thought of as tiny PV systems, reduced in space to a single point. Equipped with a battery power supply for up to 10 days, they store their data onsite with a temporal resolution of 10 Hz (averaged to 1 Hz during postprocessing). A GPS-based clocked control is used to ensure synchronization between sensors, and for proper positioning data.

The first field campaign with these instruments took place near Jülich, Germany (50.9∘ N, 6.4∘ E), from 2 April to 24 July 2013. It featured a total of 99 pyranometers deployed over an area of about 80 km2. The second was conducted near Melpitz, Germany (51.5∘ N, 12.9∘ E), and lasted from 3 September until 14 October 2013. During this time, 50 pyranometers, all of which had previously been used in the Jülich campaign, were deployed over an area of about 4 km2. During the measurement campaigns each instrument was subject to regular weekly maintenance. This maintenance included data transfer, battery replacement, thorough cleaning of the glass dome, and re-leveling of the mounting platform (if necessary). As part of this process, the states of cleanliness and orientation were recorded, in order to facilitate identification of periods of bad data. For each observation of tilt or fouling, that week's worth of data was flagged accordingly, even though the specific problem did not necessarily last for the entire preceding week.

The geometry of the pyranometer locations for the Melpitz and Jülich campaigns, as well as a histogram of all sensor pair distances dij is presented in Fig. . The sensor layout of the Melpitz campaign, with many sensors concentrated in the center and fewer towards the edge of the domain, is more structured than that of the Jülich campaign. This difference is due to the much larger spatial domain in the Jülich case, which entailed external restrictions on the instrument locations, such as road access, setup permission, and agricultural land use. Consequently, most of the very short sensor pair distances (dij<1 km), and some intermediate distances (1km<dij<2 km) are attributed to the Melpitz campaign, while sensor pair counts with dij>3 km are all associated with the Jülich campaign (Fig. c).

Taking into account the final data sets' high temporal resolution of 1 Hz, along with the corresponding dense spatial coverages, the two field campaigns provide the basis for unique analyses of irradiance variability, particularly regarding potential PV power fluctuations. At the same time, the limited durations of the campaigns result in a data set that extends from mid-spring to mid-autumn, and may not be representative of other times of the year. use data from the Jülich campaign for a performance evaluation of sky-imager-based solar irradiance forecasts, and present a more detailed discussion of the campaign and the instrumentation. To the best of the authors' knowledge, no other PV-related studies based on comparably dense and high-frequency irradiance sensor networks have been published to date.

The available global horizontal irradiance (GHI) at any given point on the Earth's surface is subject to influences from both astronomical and atmospheric processes. As for the former, the apparent movement of the Sun relative to Earth gives rise to diurnal and seasonal variations in GHI. These variations are accurately predictable and not large on short timescales of seconds or minutes. On the other hand, weather-related contributions to irradiance variability are manifold and complex, and present on all timescales. For instance, the growth, motion, and decay of clouds can affect the seasonal cycle in GHI (e.g., winter tends to be cloudier than summer in mid-latitude low-lying land), and the rapid succession of sunlight exposure and cloud shadow in conditions dominated by fair-weather cumulus generates stochastic variability on short timescales (seconds–minutes) . The presence of different kinds of cloud (e.g., layer vs. convective) at different altitudes and of different composition (e.g., high-albedo small cloud droplets vs. low-albedo large droplets in rain clouds) result in a complex set of influences on GHI over a broad range of timescales.

In order to distinguish the cloud-induced fluctuations from the slowly evolving, astronomically determined apparent motion of the Sun, a GHI time series G at a particular location may be related to either the extraterrestrial solar radiation Gextra (i.e., irradiance on Earth if there were no atmosphere), or the clear-sky radiation Gclear (i.e., irradiance on Earth with a cloud-free atmosphere). Knowing these, we can define the clearness index k=GGextra, and clear-sky index k*=GGclear, respectively. The extraterrestrial solar radiation Gextra depends only on astronomical relationships, whereas a calculation of clear-sky radiation Gclear requires parameters of atmospheric conditions, such as typical water vapor concentration and aerosol load. As the exact characteristics of Gclear are model dependent, there is no single unique clear-sky index. The use of any clear-sky model thus introduces new uncertainties to the k* time series which are absent from the original irradiance measurements. In spite of these uncertainties, use of the clear-sky index is convenient because of the pronounced non-stationarity of the irradiance time series.

While all atmospheric influences on GHI variability are included in k, variations in the clear-sky index are dominated by changes in cloud cover. Other sub-daily variations, especially those caused by changes in light scattering with solar angle α, are ideally removed entirely from k* (although depending on how accurately the atmospheric conditions and their variability are actually estimated, such changes with α may still affect the clear-sky index to a minor degree). With k* thus being better suited to remove trends in GHI variability, the focus of this analysis will be clear-sky irradiance time series computed for the respective locations of both field campaigns, using the clear-sky model described by . A limitation of this model is that it is based on climatological means and does not account for all variations in scattering or absorption properties. Moreover, relatively low values of GHI occurring shortly after sunrise and just before sunset, coupled with path prolongation and corresponding higher uncertainties in clear-sky irradiance calculations at these times, can result in unrealistic values of k* . In consequence, we only consider data associated with α>15∘ throughout the study. While the resulting clear-sky index remains an approximate model-based quantity, rather than a direct measurement, it allows us to focus directly on weather-related variations in surface irradiance.

The lowest values of k* are typically not zero, because even the darkest of clouds do not attenuate all irradiance. Additionally, the upper limit can exceed 1, primarily due to short-term reflections from the sides of clouds (and also to a secondary degree due to the limitations of the clear-sky model). Under broken cloud conditions this phenomenon, known as cloud enhancement, can cause single-point GHI to exceed its corresponding clear-sky irradiance value by more than 50 % on short timescales .

To characterize the modulation of k* variability by the prevailing sky type (e.g., overcast vs. clear sky), we divide the time series at each sensor into non-overlapping 15 min windows. This sub-hourly timescale is short enough that it is typically dominated by a single sky type, but long enough that there is enough variability to make statistical analyses meaningful. We will use differences in the statistics of k* within these 15 min windows to define different sky type categories.

To illustrate the wide range of cloud influences on k* statistics in these 15 min windows, Fig.  presents three distinct examples of spatiotemporal variability in k*. These representative subsets have been manually selected from a pool of random windows sampled from the entire duration of the Jülich campaign. Each panel includes summary statistics for all sensors in the domain for the period (represented as box plots), as well as the variability of a single randomly selected sensor. The box plots each consist of a lower “whisker” wlow, the first quartile Q1, the median Q2, the third quartile Q3, and an upper “whisker” wup (summarized in Table ). Following common practice when presenting box plots, wlow (wup) is defined as the lowest (highest) data point that still falls within the range of Q1-1.5⋅IQR and Q3+1.5⋅IQR, with IQR denoting the interquartile range IQR=Q3-Q1 . Any data below wlow or above wup are considered outliers.

The 15 min window in Fig. a features very little spatial variability and a continually low range of k* values, corresponding to a time of overcast conditions during which a fairly homogeneous cloud layer spanned the entire domain. In contrast, the majority of sensors in Fig. b show a continually high range of k* with little spatial variability, with the exception of some pronounced rapid and short-lived decreases in k*. Clear-sky conditions dominated the domain at this time, with occasional short-duration shadows cast on single sensors (although not the single example sensor). Finally, the data shown in Fig. c display considerable variability throughout the domain at all times, with a consistent IQR value of ∼ 0.5. The trace of the example sensor clearly illustrates the predominant condition of mixed skies in this case, with an alternation between cloud coverage and clear-sky exposure. The characteristic differences in the temporal average and variability of k* evident in these example data sets indicate that a natural classification scheme for sky type within 15 min can be developed in terms of the statistics of k*.

In order to assess the character of irradiance variability conditioned on sky type, we group subsets of similar sky conditions by means of two simple statistics. Specifically, we compute the sample arithmetic mean ki*‾=1N∑t=1Nki*(t), and the sample standard deviation σik*=1N-1∑t=1N(ki*(t)-ki*‾)2 of ki* (the clear-sky index of the ith sensor) for each sensor for all 15 min periods, using non-overlapping windows of width T=900 s (resulting in a sample number N=900 for the 1 Hz data).

These two statistical measures allow an intuitive characterization of the prevailing sky type that a sensor has been subjected to for a limited time, by quantifying the average and spread of the respective 15 min window in its time series. A kernel density estimate (KDE) of the joint probability density function (PDF) of ki*‾ and σik* using data for all individual sensors and all available days from both measurement campaigns is presented in Fig. . The estimated PDF is overlaid with a regular grid that we will use to define particular sky type classes.

In the low variability range σik*<0.09, the two peaks in the joint PDF (located in A1/B1, and D1/E1 of Fig. , respectively) clearly represent the cases of predominant overcast (low ki*‾ and low σik*), and clear-sky (high ki*‾ and low σik*) conditions. In addition to these two well-defined end members, intermediate sky conditions were also recorded by the single sensors. While these span the entire ranges of ki*‾ and σik*, they are not uniformly distributed. In fact, a separation between relatively clear-sky and overcast conditions (i.e., high and low ki*‾) can be observed for most values of σik*, though the distinction becomes less pronounced with increasing σik*.

The quantities ki*‾ and σik* provide single-point temporal statistical information about variability in k*. As a first characterization of the spatial structure of the k* field, we compute the spatial two-point correlation coefficients between sensors i and j

ρijk*=∑t=1N(ki*(t)-ki*‾)(kj*(t)-kj*‾)∑t=1N(ki*(t)-ki*‾)2∑t=1N(kj*(t)-kj*‾)2 conditioned on the 15 min intervals falling within individual boxes in Fig. . That is, all 15 min time windows, during which both sensors in a pair are simultaneously associated with the same sky type, are used to calculate a single ρijk* value for the respective pair of sensors. In Eq. (), ki*(t) and kj*(t) are the two individual time series of the pair of sensors for those time periods in which the statistics of both sensors are in the same grid box, while ki*‾ and kj*‾ are the corresponding arithmetic means. The quantity N denotes the number of data points in the two time series. Note that the meaning of the overbar, denoting the arithmetic mean, is different than that in Eqs. () and (), as the averaging time increases from 900 s in Eq. () to the total time of simultaneously available 1 Hz data of i and j in Eq. ().

The resulting distributions of spatial autocorrelation functions ρijk* are shown as functions of sensor pair distance dij in Fig.  for each of the grid boxes from Fig. . The same box plot statistics listed in Table  are computed for 10 logarithmically spaced bins of dij. A pair of sensors is only included in these calculations if its members share the same sky type for at least 60 min over the observational period. The number of pairs used to derive the box plot information is given in each panel.

Pairs of sensors with very high σik* and very low ki*‾ (panels A3 through A5), are found to be virtually non-existent, although these ranges are occupied by individual sensors (cf. Fig. ). Similarly, combinations with relatively high σik* and moderate or large ki*‾ (panels B4/5 and E4/5) also lack a high number of available sensor pairs. The remaining well-sampled grid boxes all show spatial autocorrelation functions ρijk* that decrease with increasing dij, as expected. However, the rates of decrease vary considerably across the different grid boxes. The differences in autocorrelation structure between two adjacent grid boxes (e.g., A1 and B1) are generally small, but become more pronounced when comparing those farther apart (e.g., A1 and D4).

For further analyses, and consistent with the manually selected exemplary periods previously shown in Fig. , we consider a classification of three distinct sky types based on the grid boxes:

overcast (A1 and B1),

PDFs of Δkτ* estimated from data from the Jülich campaign for three short-term time lags τ=(1,10,60) s are presented in Fig. . The results are first conditioned on the sky type classification (first three columns) and then shown again for all sky types together (rightmost column). As a first illustration of the effect of spatial averaging on fluctuation statistics, the PDF of area-averaged increments is also included in each panel.

All PDFs are characterized by a narrow central peak – corresponding to a high probability of very small increments – surrounded by broad tails in which the PDF decreases slowly. With increasing τ, these tails become flatter and the probability of very large excursions increases. While the PDFs of single-sensor observations all exhibit such tails, these features are much less prominent for the spatially averaged k* increment distributions. This result is consistent with previous findings of e.g., , , , and . Comparing the distributions of increments from multiple pyranometers and PV plants of different capacities for various temporal resolutions, these previous studies all showed high magnitude changes with increasing time lag, and fewer high magnitude changes with increasing PV plant size (or numbers of sensors).

Under overcast conditions, the central peaks of the distributions are generally prominent and the tails are not particularly pronounced. The PDFs of clear skies also have a strong central peak but display broad flat tails (higher than for overcast conditions), representing the rare large excursions evident in Fig. . The central peak is wider under mixed conditions, and the flanks are flatter than for clear skies. Under mixed conditions, probabilities of large excursions are enhanced by the rapid changes associated with passing cloud edges. This distribution of increments is consistent with the individual time series shown in Fig. . The statistics obtained when all sky types are taken together feature the same shapes in the tails as those of the mixed sky type, because the extreme fluctuations in k* are most common under mixed conditions.

A measure of the extent of the tails of the PDF is the probability P(Δkτ*=±0.5) of a single sensor to fluctuate by ±0.5. The values for this quantity, quoted in each panel of Fig. , take different orders of magnitude among the different sky types. These probabilities increase from overcast to clear and then to mixed sky conditions, while the values associated with the overall statistics of all sky types are located somewhere in between the last two classifications. Compared to the statistics of all sky types, and independent of τ, P(Δkτ*=±0.5) is more than twice as high under conditions classified as mixed. Thus, for applications such as the maintenance of grid stability, where worst case scenarios (in terms of strong short-term PV fluctuations) are of interest, the conditioning of Δkτ* statistics on different sky types demonstrates the strong dependence on specific sky conditions of the likelihood of severe fluctuations occurring shortly one after another.

While changes in increment variability are reflected by changes of σΔkτ* (dashed lines in Fig. ), the standard deviation is not an appropriate measure of the size of extreme fluctuations due to the non-Gaussian character of k* increment statistics. In consequence, the widely used three-sigma rule of thumb, according to which a range of Δkτ*‾±3σΔkτ* would cover 99.73 % of the values if Δkτ* were normally distributed , can be misleading when applied to k* fluctuations. For example, only about 95 % of the empirical single-sensor data are included in this range for the most variable subset of τ=60 s and mixed skies (Fig. k). This result is in line with previous findings of the 99.7th percentile of 1 min increments in clear-sky index being about 7 standard deviations away from the mean .

The differences between the single-sensor increment statistics and the distributions of areal averages in Fig.  are of interest, because the spatial averaging clearly results in a substantial reduction in the probability of high-magnitude k* fluctuations. In order to specify the underlying spatiotemporal field characteristics, we analyze two-point correlation coefficients of k* increments ρijΔkτ* (computed as in Eq. ), based on data from both field campaigns. Various models have previously been proposed to predict the behavior of increment correlation as a function of distance for specific temporal scales, either by using empirical fits to measured data or by modeling simplified cloud shapes . While the empirical fits are based on data sets of limited spatiotemporal resolutions, the more theoretical model does not account for the complexity encountered in real cloud shapes. Our data set permits a direct empirical assessment of the spatial correlation structure of increments.

Conditioned on the previously defined classification scheme of sky types, summary statistics of ρijΔkτ* (cf. Table ) are presented as functions of sensor pair distance for different time lags in Fig. , using 10 logarithmically scaled dij bins. Also shown in Fig.  are the autocorrelation structures as functions of sensor pair distance divided by increment size, obtained using Δkτ* time series of all three increments τ=(1,10,60) s. The plots only include those sensor pairs whose members simultaneously correspond to the same sky type for a total of at least 60 min.

A useful measure of increment correlation structure is the decorrelation length scale d0, which we define to be the minimum distance at which ρijΔkτ*=0.05. These distances generally increase with increasing time lags, and they decrease from overcast to clear skies, and then to mixed conditions. As an exception to the latter statement, under overcast and clear conditions k* increments associated with very short time lags τ=1 s (panels a and b) are uncorrelated even for very small dij. The mixed sky type features a measurable decorrelation length scale d0≈90 m for τ=1 s. For τ=10 s (τ=60 s), this distance increases to 2.2 km (7.2 km) under overcast, 0.2 km (1.5 km) under clear-sky, and 0.2 km (1.1 km) under mixed sky conditions. These decorrelation distances estimated from Fig.  are all summarized in Table .

In accord with previous studies published by and , increasing time lags are accompanied with increasing correlation coefficients for any given dij<d0. Additionally, ρijΔkτ* decreases with increasing dij for any fixed τ, converging to zero. Both of these earlier studies analyzed lags on distances between tens and hundreds of kilometers. The analysis uses 1 min averages of k* from 23 pyranometers, while derive their results from hourly satellite data. As anticipated by , the presence of negative correlation extrema found during their single-sensor virtual network study are not observed in our analysis of observationally based ρijΔkτ*. The transitions of the spatial autocorrelation structures of Δkτ* between sky types and timescales shown in Fig.  in part resemble those presented by , who analyzed the relationship between correlation, distance, and timescale of PV power fluctuations, using data from 70 DC/AC inverters in a multi-megawatt PV plant.

In addition to the statistics of the measured data, each panel of Fig.  also includes colored regions corresponding to the model of spatial correlation coefficients ρijΔkτ*=(1+dijτ⋅v)-1 proposed by , using a range of cloud speeds 2 ms-1 < v<10 ms-1. These speed values correspond approximately to the range of mean vector winds at 850 hPa v850 for Germany from the ERA-40 re-analysis atlas 2 ms-1 ≲v850≲ 8 ms-1;. While obtained Eq. () as a curve fit from data with hourly resolution and pair distances up to several hundred kilometers, they also considered an initial assessment of its applicability to higher spatiotemporal resolutions. Compared to other models, showed Eq. () to yield relatively high correlation coefficients and relatively long decorrelation length scales. Of the models considered by , the selected one provides the best fit to our data. As the model makes no distinction between sky types, its output in Fig.  is the same for each row of panels and the quantity v cannot be interpreted as the actual speed of the clouds. Rather, it represents a quantity with the units of speed that combines information about cloud type and motion.

The ranges of the model results are in broad agreement with the summary statistics of the two field campaigns and the general decrease of spatial correlation with increasing distance is reproduced well. However, differences between overcast and clear-sky conditions are evident, as the former tends to coincide with the upper region of the model range (corresponding to high v), while the latter rather agrees with the lower end of the range (low v). Overcast conditions for τ=1 s again are an exception to this general rule. For the mixed sky type, and for all values of τ, the correlation values coincide with the upper region of the model for very small dij, but they decrease more rapidly with increasing distances and thus feature decorrelation distances that are consistently much smaller than the modeled ones.

Finally, the bottom panels (Fig. j–l) bring the above results together by presenting the correlation coefficients of different sky types as functions of distance over time lag. Again, overcast conditions coincide with model outputs for high values of v almost entirely, while results for clear skies mostly overlap with modeled correlation coefficients for low v. Mixed conditions are highly correlated for short scaled distances (corresponding to the upper region of the model), but de-correlate rapidly with increasing dij⋅τ-1. If the range of v values considered was broadened, the correlation structures would fall within the model envelope for all sky types. However, while the profiles for overcast and clear skies look similar to the model prediction for some specific v, this is not the case for the mixed cloud case. For short scaled distances the correlation decay corresponds to an intermediate value of v in the model – but after that, the correlations drop off much faster. This result demonstrates that the model is not able to capture the correlation structure for mixed sky conditions.

The pyranometer network measurements are hosted on the HD(CP)2 data portal, which is accessible via the project website (http://www.hdcp2.eu).