During thunderstorms, LOFAR dumps about 2 s of raw-voltage (transient buffer) data from 6 low-band antennas (LBA) per station, from 38 Dutch stations. This data dumping takes about 15–30 min, thus LOFAR can record up to 10 flashes per storm. The LBAs are inverted V dipoles that are sensitive to 10–90 MHz. After recording, the data is processed into images with the “impulsive imager”. This is an advanced time-of-arrival algorithm inspired by Kalman filters using cross-correlations to measure time-delays of pulses received by different antennas . The impulsive imager produces about 200 sources per ms with meter and nanosecond level accuracy.

define sparkles as “spatially small (not exceeding a few 100 m) intermittent discharges near the top of the clouds where this activity is seen over extended periods lasting the full duration of a LOFAR recording.” An example of a LOFAR image with sparkles is given in Sect. .

In order to perform a quantitative analysis, we distinguish sparkles from other lightning structures using a two-stage approach. Inspired by , we apply the DBSCAN (Density Based Spatial Clustering with Noise) algorithm as implemented in Scikit-Learn  to group VHF sources into four-dimensional lighting structures. We used visual inspection and a trial-and-error approach to find appropriate normalization constants for clustering. The normalization constants determines if two VHF sources are far apart (larger than the normalization constants) or near each other (smaller than the normalization constants). The final parameters for the two-stage clustering approach are given in Table .

In the first stage, we normalize the time and spatial coordinates of all VHF sources by dividing by normalization constants of 0.3 s and 2 km respectively. We then run DBSCAN with a minimum cluster size of 30 points, classifying each resulting cluster as “other VHF sources”. Unclustered points, which are apparently not part of any large cluster, proceed to the second stage.

In the second stage, we re-normalize the original time and space by dividing by 0.02 s and 200 m respectively, and now use DBSCAN with a minimum cluster size of 2 points. These clusters are classified as sparkles. Any remaining VHF sources, which were thus potentially part of sparsely imaged large clusters, are added to the “other VHF sources” category.

Finally, we reassign any sparkle VHF source to the “other VHF source” category if it is located below an altitude threshold of 8 km. This threshold removes a small number of outliers below this threshold.

The result of the sparkle classification algorithm as described above, is a categorization of each LOFAR VHF source as either a “sparkle”, or as an “other VHF source”.

By design, the VHF sources that are marked as sparkles only entail relatively small isolated clusters. It is likely that needles, which connect to decaying positive leader channels, are not classified as sparkles.

The radar data that is used in this study comes from a polarimetric C-band radar, located on the island of Borkum in Germany. This specific radar was chosen given its favorable location and range with respect to the location of interest, i.e. the north of the Netherlands. The technical details of this radar can be found in . The key aspects of the Borkum radar and data can be found in Appendix .

Throughout the analysis, the LOFAR data is compared spatially with radar data. This process involves a standard correction for the refractive index of the atmosphere , georeferencing and advecting the radar data according to the ambient wind velocity. These steps are elaborated on in Appendix . Note that all radar images in this work are advected to the time as indicated in the figure caption.

In this study, we compare radar data within a 2 km radius of VHF sources identified as sparkles (Sect. ) with radar data near other VHF sources. The latter category is radar data found within a vertical column of 2 km radius of other VHF sources, but not within a 2 km radius of sparkles. A minimum altitude threshold of 8 km ensures a fair comparison of radar variables, without any potential altitude bias. A Zh>0 dB threshold ensures that we only use radar data with significant amounts of hydrometeors. Furthermore, the 0 dB threshold ensures that the signal is significantly above the noise level. Thus we do not consider the atmosphere outside clouds and clouds that are barely reflective to the Borkum radar.

We acknowledge that there may be errors in the advection of radar data (see Appendix ). Additionally, the coarse cross-beam radar resolution may obscure vertical gradients and therefore potentially bias the radar statistics. For some VHF sources more than 100 km away from the radar, the vertical radar resolution amounts to ∼ 4 km. In this light, it is a deliberate choice to take a 2 km radius to select radar data near VHF sources. At the same time, this coarse radius allows us to collect enough statistics. In a sensitivity analysis (not shown), changing the radius to 1 to 3 km did not lead to notably different results.

We compare the distributions of Zh and Wrad values measured near sparkles with those measured near other VHF sources. To quantify differences between distributions we compute the two-sample Kolmogorov–Smirnov (KS) statistic and its p-value , together with Cliff’s delta as an effect size . The KS statistic measures the maximum absolute difference between the two empirical cumulative distribution functions; larger values indicate greater discrepancy between distributions. The KS p-value tests the null hypothesis that the two samples were drawn from the same distribution. One can only conclude a significant difference between distributions, if the KS p-value is low – for example, below 1 %. Cliff’s delta is a nonparametric effect-size that represents the probability that a randomly chosen observation from one sample exceeds a randomly chosen observation from the other, minus the reverse probability; values range from -1 to 1, with magnitude indicating the strength of the difference.

In Fig. a the sparkles are relatively easily distinguished as a confetti-like cloud between 8 and 12 km. Using the the algorithm and parameters described in Sect. , we find that the confetti cloud is classified as sparkles, while obvious large lightning structures are marked as other VHF sources. This is illustrated in the figures in Sect. S3 in the Supplement.

Figure  provides a zoom-in of the LOFAR data from Fig.  with the sparkles indicated as triangles with black outline. We note that there are small clusters that are not classified as sparkles. Apparently, these clusters contain more than 30 VHF sources, or multiple small (<500 m) clusters are grouped as a single large structure by the classification algorithm. In general, this type of small clusters in the “other VHF” category, is located less than 2 km distance from clusters in the sparkle category. Taking into account the 2 km radius to select radar data near sparkles and exclusion from the subset of radar data near other VHF sources, we expect that misclassifications into the “other VHF” category will only have a marginal effect on the results.

By design, the second stage of clustering in the sparkle classification removes isolated single VHF sources from the sparkle category. We observe three types of isolated VHF sources: (1) isolated VHF sources at low altitudes <5 km that seem to be part of larger, sparsely imaged lightning structures; (2) isolated VHF sources part of dart leaders; and (3) isolated VHF sources in or near the confetti cloud of sparkles. The latter category may be poorly imaged sparkles and can therefore be misclassified as other VHF source. However, given the 2 km radius to select sparkle data, we expect that these misclassifications will only have a marginal effect on the results. The VHF sources at low altitudes and in dart leaders (observations (1) and (2)) are rightfully classified as other VHF sources.

On 18 June 2021, the atmosphere was very unstable. According to ERA5 data , around 18:00 UTC (all times from here on in UTC) there were places in the Netherlands where CAPE (convective available potential energy) values exceeded 3000 J kg⁻¹. At times, the shear and helicity values exceed 20 and 200 m² s⁻² respectively, for example in Fig. . These values are favorable for the formation of supercells. Supercells are a type of convective system characterized by a single very strong updraft, mesoscale rotation in the updraft, and overshooting cloud tops . Much turbulence is expected around and aloft the updraft, particularly in the overshooting cloud tops. Supercells typically produce much precipitation, large hail and much lightning .

In the domain of the LOFAR images, there are places until at least 20:00 where CAPE exceeds 2000 J kg⁻¹. The CAPE, shear and helicity values decrease significantly after 20:00, after which conditions become less favorable for any convection to organize into mesoscale structures.

In order to match the LOFAR data with the storm dynamics, we distinguish five different convective systems in the northeast of the Netherlands between 17:00 and 21:00. The different systems are labeled as “A” through “E” in Fig. . The convective systems are primarily inferred from radar reflectivity (Zh) values. The approximate location of lightning as imaged by LOFAR is indicated by black (sparkles) and dark blue (other VHF sources) contours. The locations of sparkles follow from the sparkle classification algorithm as described in Appendix . In the following, we give give a brief description of the convective systems.

Figure  shows histograms comparing the Zh and Wrad radar data near sparkles to radar data near other VHF sources. The Zh and Wrad values are very different near sparkles. The average Zh and Wrad values are higher (see Table  and black lines in Fig. ) for sparkles. The distribution of sparkles also show a heavier tail for higher values. KS p-values are well below 1 %, indicating a statistically significant difference between sparkles and other VHF sources. Sensitivity tests (see Sect. S2 in the Supplement) show that these results are not sensitive to varying the radius to collect radar data near VHF sources, or varying the parameters for sparkle classification. Comparing the radar data at different altitudes (see Sect. S2 in the Supplement), the Zh values are significantly higher at all altitudes. For Wrad, the values are only significantly higher below 12 km altitude.

Figure  shows histograms of Zh versus Wrad, allowing to investigate their correlation. In the panel b and d, where the number of counts are normalized per Zh bin (vertical columns), we observe a nonlinear correlation. Panel f, the difference between panel b and d, shows that the Wrad values are consistently higher for values of Zh<50 dBz. The KS statistic and Cliff's δ per Zh bin are provided in Appendix .

Figure  shows the results of the HMC algorithm. For sparkles (red), a relatively larger percentage of radar data is classified as graupel/hail (GH). Near other VHF sources (blue), more data classified as ice crystals or dry snow. Figure b and c can be used to estimate the quality of the HMC algorithm. The performance of the HMC algorithm for each category can be estimated both with the maximum mean probability for each row, always found on the diagonal, and with the difference between the probabilities on each row. A larger difference between the maximum and other probabilities, indicates a better classification. The results show that the the “Graupel/Hail” and “Hail” classification are relatively certain, with a difference of ∼ 0.3 between the maximum and second highest mean probability. In contrast, the HMC algorithm has much more difficulty in distinguishing between “Dry Snow”, “Horizontal Ice Crystals”, “Vertical Ice Crystals” and “Light Rain” categories. These categories have a difference of ∼ 0.1 between the maximum and second highest mean probability. Regarding the orientation of ice crystals, we see relatively more vertical aligned ice crystals (VC) for sparkles. The opposite holds for other VHF sources.

In order to understand the quantitative results in Sect. , we have also inspected the individual radar images. Figures , and show radar images with an overlay of contours to mark data corresponding to “sparkles” and “other VHF” sources. Note the threshold of 8 km altitude and the 0 dB reflectivity that is used to select radar data. Figures for additional radar elevation angles, for system A and B at times corresponding to LOFAR observations can be found in Sect. S4 in the Supplement.

A first observation is that the radar data near sparkles is always a subset of the radar data near other VHF sources. Although it is possible that large lightning structures exist in the same volume as sparkles, it is also possible that this is an effect of an incomplete sparkle classification algorithm. As mentioned in Sect. , incomplete sparkle clustering should have marginal effects on the statistics presented in section , because the subset of radar data near “sparkles” is excluded from the radar data near “other VHF” sources.

In general, the radar images often show high Wrad values where the radar beam exits a precipitation cloud at high altitudes. These areas often have low Zh values and they likely mark the interface between the stratosphere and precipitation cloud. The cloud-stratosphere interface with low Zh and high Wrad values are seen in the leftmost columns of Fig. e and f.

As expected from the histograms in Fig. , we see that the sparkles are often found in areas with high reflectivity values. We also note that within the sparkles data (black dotted contour), the radial velocity Vrad shows much divergence and rotation. This is seen in the images as the radial and cross-radial gradients in the Vrad values respectively. The images show that locations with rotation and divergence are often accompanied by local points of relatively high Wrad values.

In Figs.  and we observe that sparkles mainly surround the location of an intense, isolated updraft column. The approximate location of the updraft column at ∼ 10 km is indicated by a white ellipse. This ellipse marks a patch of positive Vrad values (∼ 1 km²) surrounded by negative Vrad values. Such a Vrad pattern could be caused by (1) high vertical velocities in the updraft that overcome hydrometeor sedimentation, (2) mesoscale rotation of an updraft, or a combination of (1) and (2). The intensity of the updraft is highlighted by the observation that the Zh values are > 40 dB at the location of the positive Vrad patch. Only a strong updraft would offset the large sedimentation velocities that are associated with Zh>40 dB values. We also note the relatively high Wrad values around the updraft column.

The location of the Zh core in Fig.  is displaced ∼ 0.05° from the location of the updraft at ∼ 10 km (white ellipse). However, this displacement is roughly the same throughout the whole convective system. Therefore the displacement is probably related to advection between the different radar scans and the slanted orientation of the updraft columns. In Fig. , aliasing of the Vrad signal gives ambiguous rotation and divergence in the updraft column. However, we do clearly see local points of enhanced Wrad values for the patch of sparkle data.

For Fig. , one can use similar observations of positive Vrad patches where Zh values are high to find two or three updraft columns. However, these locations are not so isolated as the updraft seen in Fig. . The locations are not indicated.

In accordance with the statistics in Fig. , the radar images show that locations of sparkles are correlated in regions that have been classified as graupel or hail (GH). Especially at high altitudes, for example in Fig. , we see that the sparkles data is more limited to areas that have been classified to feature graupel or hail (GH), whereas the other VHF data encompasses vast areas that have been classified to feature ice clouds (HC and VC).

Given the clustering parameters as presented in Sect. , the sparkle classification algorithm yields satisfactory results. Given the proximity in space, we acknowledge that some small clusters (e.g. Fig. ) and some isolated VHF sources may be misclassified as other VHF sources. We expect that the impact of such false negatives is marginal. Considering the 2 km radius to select radar data, many of the false negatives would be masked by the proximity of sparkles. We also note that changing the algorithm to reduce false negatives, for example by increasing the minimum number of VHF sources for a large cluster or by omitting the second stage of small-scale clustering, leads to more false positives. This is unfavorable for the selection of radar data, because the 2 km radius near sparkles gets priority over the 2 km radius near other VHF sources.

show that some small‐scale discharges are similar to so called “needles” discharges, and are related to positive‐leader breakdown. Our two‐stage clustering (first stage temporal margin t=0.3 s) is likely to merge such rapid, localized needle pulses, and any accompanying dart leaders, into larger clusters. Consequently, our results only represent sparkles that are not clearly linked to positive‐leader breakdown.

Although the classification for graupel/hail (GH) is relatively certain, panel b and c in Fig.  show that the distinction between horizontal crystals, vertical crystals, dry snow, and light rain is more uncertain. Therefore, we can not draw substantiated conclusions about any electric field effects on ice crystal alignment. On the other hand, we are confident that the surroundings of other VHF sources are often filled with ice crystals. For sparkles, the surroundings are dominantly filled with graupel/hail, but almost equally much space with ice crystals. That is, if we aggregate the horizontal and vertical ice crystals into one hydrometeor category.

Another aspect to consider for the fuzzy-logic algorithm is that it was originally designed for S-band radars. Because of their larger wavelength, S-band radars are less prone to Mie scattering and attenuation . Although previous work has shown the potential of fuzzy-logic algorithms for C-band radars, attenuation and Mie scattering remain a source of error. The algorithm we use in this paper does compensate for Mie scattering, but no attempt is made to compensate for attenuation. For improved results, one may attempt a more complicated scheme to weight the different polarimetric variables, following .

Placing our results in the context of previous work, the following similarities are found. The correlation between high reflectivity values and sparkle-like lightning is also found by , and . These studies also observe that these discharges form a cap on top of a very strong, vertical updraft, reaching high altitudes. Our study also suggests that sparkles are found at the top or above strong updrafts, based on three arguments. Firstly, the high reflectivity values at sparkle altitudes can logically only be present in or near updrafts. The large hydrometeors responsible for such high reflectivity values would quickly sediment down to lower altitudes were it not for the presence of strong updrafts. Secondly, the Vrad radar images indicate updrafts. For both system A and B, the images (see Sect. S4 in the Supplement) at low elevation angles show mesoscale rotation, indicated by cross-radial Vrad gradients, at locations with high Zh values. For the radar image in Fig. , we even see a patch of positive Vrad values at about 10 km altitude, well above the low-level rotation. This indicates that the updraft extends up to more than 10 km altitude. Figure  shows the location of sparkles, just above this location. Thirdly, the relative location of the BWER and hook echo in system A indicate that the main updraft is much aligned with sparkles, especially when one considers that the updraft is likely to be slanted by wind shear. Unfortunately, time differences up to 2.5 min between radar and LOFAR data, broadening of the beam width, and the few radar elevation angles reaching sparkle altitudes (see Appendix ) inhibits the quantification of the distance between sparkles, updrafts, and cloud tops.

All of the aforementioned studies also specifically associate the sparkle-like discharges with the intensification of updrafts. The ∼ 20 min gap between LOFAR images complicates a detailed temporal analysis of sparkles. However, if the convective systems that produce sparkles are indeed a supercell and a squall line, it may be expected that updrafts were intense. The story line of the mesoscale events in convective system A, such as the development of a BWER and hook echo, would suggest intensifying updrafts around the time of sparkles. For convective system B, there are few mesoscale features that indicate the updraft intensification matching the presence of sparkles. However, we think that attenuation obscured some of the dynamical features of the right-hand-side of system B (see Fig. ), which produced many sparkles.

We are confident that for high Zh values, the higher Wrad values near sparkles are mainly reflecting the enhanced atmospheric turbulence near sparkles. Turbulence enhances Wrad through wind-shear within the radar resolution volume (sub-resolution). Not only does wind shear drive turbulence, but the chaotic fluctuations of wind shear also constitute turbulence. Qualitative comparison shows that, for higher reflectivity values, enhanced Wrad values are generally accompanied by wind shear between radar resolution volumes. Thus, the driver for sub-resolution turbulence in cloud tops is clearly visible. Examples can be seen in Figs.  and as spatial gradients in the Vrad image, at locations where both Zh and Wrad values are high.

For low reflectivity values, it is likely that side-lobe contamination plays a role in enhanced Wrad values. This effect is most pronounced if, the Zh value for a specific radar resolution volume is not high, bu Zh values are high in the surroundings. In this case, the radar beam's side-lobes, which also probe the volume outside the targeted resolution volume, contribute disproportionally to the total returned signal. Because a larger volume contributes to the total signal, measured wind shear, and thus Wrad values, can increase. Examples of locations that are likely to be the result of side-lobe contamination, are marked with the arrows in Figs. –.

Side lobe contamination explains the high Wrad values observed in Fig.  for the lowest Zh bins. From Fig. f, it appears that the side-lobe contamination is more prominent for sparkles. This can be expected because the sparkles are, more often than other VHF sources, concentrated near highly reflective updraft columns. It is near these updraft columns that the spatial Zh gradients, and thus side-lobe contamination is largest. Since side-lobe contamination is most significant for low Zh values, we are confident that the observed Wrad enhancement actually represents atmospheric turbulence for Zh>15 dB.

Another mechanism that can increase Wrad values, is differential sedimentation of mixed hydrometeors. However, because the radar elevation angles are relatively low (≤25°), the sedimentation velocities, which are inherently vertical, are a minor contribution to the spectrum of the velocity in the direction of the radar beam. Additionally, this type of enhancement is most pronounced when the backscattered signal is not dominated by a single type of hydrometeor. The HMC algorithm shows, comparing the numbers between the rows in Fig. , that the uncertainty in the classification is similar for both sparkles and other VHF sources. Therefore, we conclude that for sparkles, there is no increased hydrometeor mixing and thus no enhanced Wrad through differential sedimentation.

The presence of intense turbulence near sparkles, as suggested by the enhanced Wrad values, may be important for the relevant charging mechanisms that explain the presence of these small-scale lightning structures.

Non-inductive charge separation is generally considered to dominate the charging mechanism in thunderclouds. However, the inductive charging might be considerable in certain conditions. showed that for turbulent conditions, an ambient electric field and hydrometeors of adequate size, the inductive charging mechanism may lead to exponential growth of the electric field. Supposing that the local spots of enhanced Wrad values represent the fine scale turbulence needed for significantly enhance the inductive charging mechanism, this could explain the intermittent behavior of sparkles that is observed by LOFAR.

In light of the small spatial extent of sparkles, we evaluate mechanisms that could form charge pockets and thereby limit growth of sparkles. Modeling work by shows that charge pockets may be related to heterogeneous charging in the updraft column. Turbulence in cloud tops could fragment those pockets down to the ∼100 m length scale associated with sparkles. Our results do not clarify the role of heterogeneous updraft charging. However, given our observation of increased turbulence near sparkles, turbulent fragmentation seems plausible. The size of the charge pockets in the study from do match well with the size of sparkles. They show that small charge pockets are abundantly generated at the top of the updraft of a supercell. The main mechanism for the small extent of charge pockets are turbulence with length scales > 250 m (resolved in their simulations). However, the unresolved (< 250 m) turbulence can also play a significant role for further fragmentation of charge pockets. The increased turbulence that we observe in the vicinity of sparkles and their location to the updraft columns, matches well with the results from . Alternatively, some works suggest that velocity gradients at the top of overshooting clouds may cause folding of a charged screening layer into the cloud . The folding process could potentially form pockets of the oppositely charged screening layer wrapped into the cloud. If so, we would expect to see a complex Vrad pattern along the top-edge of the cloud. In Fig. b, near sparkles (black dotted contour), we indeed see a “frayed” pattern in the Vrad gradient at between 11 and 12 km altitude. Folding in cloud tops could also be related to the irregular coiled shapes in both the Zh and Vrad images near sparkles. We also note that the mechanism to drive screening layer folding, namely wind shear, may be visible as enhanced Wrad values at the top-edge of clouds. For example near the 12 km contour line of Fig. . However, we cannot distinguish tropopause wind-shear from side-lobe contamination (arrows in Figs. –).

Altogether, our observations are compatible with both hypotheses of (1) folding of a screening layer into the cloud, and (2) turbulent mixing of larger charge regions into small charge pockets. These mechanisms may act jointly.

As suggested by , the proximity of the charged screening layer to the lofted charge in the updraft might enhance the electric field. If proximity to the screening layer is indeed a condition for sparkle discharges, it could explain why turbulence in and near the updraft has enough time to form small change pockets. Our results results near sparkles, with reflectivity values and graupel/hail classification indeed suggest that the lofted charge, carried by large hydrometeors, is brought relatively high and thus close to a screening layer. However, with our coarse vertical radar resolution, we are unable to check the distance of sparkles to a charged screening layer.

This study uses a unique combination of high resolution lightning data and operational weather data. Further research with similar methods may disclose the location of sparkles with respect to a charged screening layer, charge pockets, and enhanced turbulence.

A first improvement for future studies will come with the release of LOFAR 2.0, planned for 2027. The new system can observe significantly more flashes per storm. This will allow a better comparison of the temporal development of lightning and sparkles with the thunderstorm dynamics.

It would be most useful to deploy a mobile research radar for the study of sparkles. Using more steep elevation angles, potentially with an adaptive scanning strategy that focuses on the overshooting cloud tops, would allow to get a much higher vertical resolution in the location which is most relevant for sparkles. Additionally, an extended analysis of the radar data, for example, a spectral analysis similar to , could provide more information on the mixture of different hydrometeors and the source of the observed Wrad enhancement.

Finally, the instruments aboard the Meteosat Third Generation (MTG) satellite that was launched in 2022 could provide useful information on the relation between cloud top height and lightning. In particular, matching the MTG lightning data with LOFAR lightning images may reveal whether the sparkles occur above the clouds.