Black carbon (BC) aerosols are strong absorbers of solar radiation formed from incomplete combustion of fossil fuels, biofuels, and biomass . In the atmosphere, BC is usually found together with other types of aerosols, which form a coating around it . To understand the impact of BC on the environment, global climate models require information about its light-scattering and absorption properties . The most common morphology assumed for such BC-containing aerosols in light-scattering codes is a spherical core–shell shape . The Lorenz–Mie theory is often used to calculate the optical properties of such spherical BC particles . However, studies have shown significant discrepancies in the results of the Lorenz–Mie theory when compared with ambient measurements .

High-resolution transmission electron microscopy (TEM) images showed that the BC particles have a fractal structure composed of numerous spherules known as primary particles . This led to an advanced mathematical description of BC as fractal aggregates, known as fractal law : 1Npp=kfRgaDf, where a is the radius of the primary particle, Npp is the number of primary particles, kf is the fractal prefactor, and Df is the fractal dimension. Rg is the radius of gyration, which characterizes the spatial size of the aggregate. The shortcomings of the simplified spherical assumption of BC have caused the scientific community to develop towards the use of such realistic fractal aggregate morphology for computing the optical properties of BC (e.g., ).

showed that the discrepancy between modeled and measured optical properties could be reduced to 10 % when an aggregate morphology is used. To simulate the optical properties of BC as fractal aggregates, the most commonly used methods are the Rayleigh–Debye–Gans (RDG) approximation , the discrete dipole approximation (DDA) , the generalized multi-particle Mie (GMM) method , and the T-matrix method . The multi-sphere T-matrix (MSTM) method has found widespread applications in the research field because of its high computational speed and accuracy in comparison to other methods like the DDA . Although the MSTM has lower computational costs when compared to other numerical methods, a single simulation can still take more than 24 h, depending on the properties of the aggregate.

Consequently, pre-calculated databases have been developed for aggregate properties to save time in constructing detailed aggregates and time-consuming optical simulations . Using these databases as look-up tables mitigates high computational overhead in large-scale applications. Still, this approach is limited by the range and step size of parameters chosen during the database creation. Previous work has trained machine learning (ML) models on such databases to overcome those limitations. Once trained, those ML models provide predictions for BC optical properties in a fraction of a second. trained a support vector regressor on a database generated using MSTM simulations (Npp from 8 to 3000; Df from 1.8 to 2.2). However, they did not consider coating and used pure BC aggregates in their experiments. Their results also suggest that their model has considerable difficulties when attempting to predict optical properties for physicochemical properties not in the range of the training data. predicted optical properties of uncoated BC fractal aggregate using a graph neural network (Npp from 8 to 960; Df from 1.8 to 2.3). The input graph contains one node for each primary particle and an edge between two nodes if the distance between the corresponding primary particles is less than some threshold. The authors generate their ground truth database using the MSTM algorithm, but, like , they do not consider any coating in their experiments. The machine learning methods, training parameters, performance metrics, and other details of and are compared to this study in Table .

This study demonstrates the use of a machine-learning-based approach to predict the optical properties of BC aggregates at various aging stages, including coating, which is highly relevant for atmospheric aerosols. Combining this ML-based approach with a laboratory dataset showed that optical properties like single-scattering albedo (ω) and mass absorption cross-section (MAC) can be predicted more accurately than with conventional Mie-based methods. A database of the optical and physicochemical properties of BC has been built for this study, which is an extension of the previous work by . We trained two ML methods on this database: kernel ridge regression (KRR) and artificial neural networks (ANNs). Experiments show that these models predict the optical properties of BC aggregates regardless of their size, morphology, or composition at low computational costs and with high accuracy. The dataset used to train our ML models is freely available at Zenodo

https://zenodo.org/records/7523058 (last access: 23 January 2024)

The paper is structured as follows: Sect. 2 provides an overview of the physical, chemical, and optical properties of BC used in this study. Section 3 describes the machine learning techniques, including the data processing, machine learning algorithms, and evaluation procedures. In Sect. 4, the results demonstrate that realistic morphologies of BC can be used to accurately predict optical properties at various stages of aging. Section 5 discusses how the results compare to laboratory measurements of BC, discussing the atmospheric processing in detail. Potential limitations and challenges of this work are presented in Sect. 6, and we end with the main conclusions in Sect. 7.

The database for the physicochemical and optical properties of BC fractal aggregates has been designed to consider all the possible aging stages of BC. The optical properties of BC fractal aggregates are most sensitive to the change in particle size as they age . The particle size is reported as dependent parameters of the number of primary particles (Npp), volume equivalent radii (ri and ro), and mobility diameter (Dm). Furthermore, the chemical composition and morphology also influence their optical properties. There are constants related to the particle's chemical composition, such as density and refractive index. The optical properties have been reported as efficiencies and cross-sections. Further dependent optical properties have also been included. The mass and volume of the BC particles were used for conversion between various optical parameters. Furthermore, some parameters, such as the wavelength, were related to the optical model. The database was created using 6192 particles of varying sizes, morphologies, and coating fractions. There are 35 features in the database, which are categorized into 15 physicochemical features, 13 optical features, and 7 constants. Sect.  contains an overview of all the features of the database. In Table , the upper and lower bounds of the main features are provided.

As mentioned in Sect. , several high-impact applications, such as climate modeling , depend on accurate optical properties for specific BC particles. Hence, we propose to train an ML model on a pre-computed database containing physicochemical and corresponding optical properties of BC fractal aggregates at several life cycle stages. This model will learn patterns and structures within the data and should generalize to unseen data values when used in applications, as evidenced by the success of ML in several domains . In this work, we train kernel ridge regression and a multi-layer perceptron on the database introduced in Sect. . The following sections detail our data processing routines, models, and evaluation procedures.

The error distributions for the ML methods are presented in Fig.  for different experimental scenarios of the data-splitting with respect to the parameter fractal dimension. The median error is close to zero for the random and interpolation splits, meaning our models do not generally over- or underestimate any optical value. The distribution of errors (excluding outliers) for the random and interpolation splits is relatively narrow, indicating that most test points have minor errors. In the extrapolation case, both ML models exhibit bias, such as overestimation of Qsca by ANN and overestimation of g by KRR. However, the mean absolute error, even for the extrapolation split, is 1.5 % to 8 %, which is still within reasonable limits. showed that their model has considerable difficulties when attempting to predict optical properties for parameters not in the range of the training data. However, adding a few data points to extend any parameter range significantly improved the prediction ability of the ML algorithm. The interpolation and extrapolation results are similar if training data and test data are split according to the parameters of the coating fcoating and particle size Dm. The Appendix provides a more detailed discussion about the interpolation and extrapolation results for parameters of fcoating and Dm in Figs.  and , respectively. Overall, the narrow boxplots of the errors in the random split demonstrate the effectiveness of the ML algorithms in predicting the optical properties of coated BC fractal aggregates.

The MAEs for our experiments are reported in Table . In the case of the random split, both ML models are pretty accurate, with the percentage of MAEs ranging from 0.1 % to 0.4 % when compared to the average feature range. reported mean absolute percentage errors (MAPEs) between 2 % and 9 % for their optical predictions, whereas reported relative errors between 1 % and 5 %. The MAPEs are biased to the magnitude of the true value in the denominator. The same MAE can result in a significantly different MAPE depending on the magnitude of true value they are divided with. In our view, the prediction error should be weighted equally for both points; therefore, we chose the MAE as our error metric. also discussed how the bias of MAPEs resulted in higher values of nearly 70 % for smaller particles. Error distributions for the ML methods shown in Fig.  are presented in terms of MAPE in the Supplement. The comparison of the two ML methods for random split in Table  showed that KRR generally results in a lower MAE for predictions of Qabs and Qsca. Contrary to this, ANN could predict g with a lower MAE. In line with expectations, the MAE for the splits based on interpolation and extrapolation is somewhat higher. The errors, however, are still regarded as relatively minor compared to the features' range. The extrapolation and interpolation experiments were used to test the performance of the ML algorithm under various scenarios of data available for training. The ML models we publish for use in applications were trained on the entire dataset using the best parameters from the random-split experiments. As a result, the errors should be similar to those we report for the random split here.

A one-to-one comparison was performed between the estimates and true values to understand better how the ML methods predict optical properties. Figure  compares the estimated and true values for the wavelength of 660 nm when the training and test data are randomly split. The values of Q^abs, Q^sca, and g^ obtained from the KRR and ANN methods are compared to the true values derived from the MSTM method. The performance of both ML methods was studied for BC fractal aggregates with three representative morphologies and coating fractions (Df = 1.5 and fcoating = 0 %; Df = 2.1 and fcoating = 50 %; Df = 2.7 and fcoating = 90 %). There was reasonable agreement between KRR and ANN for all sub-cases. Therefore, the machine learning models appear applicable in a broader context. The model does not overfit with different coating fractions and complex morphologies. The one-to-one comparison results agree with the results from , which also showed reasonable predictions of Q^ext, Q^sca, Q^abs, and g^ across the entire range of size parameters.

During their lifetime, BC fractal aggregates undergo complex changes in size, composition, and morphology due to atmospheric processing. Figure  shows a visualization of how the ML predictions compare to the MSTM reference for different aging scenarios for BC fractal aggregates. It compares the estimated and true values of the optical properties for the random split. The models trained using a random split of training data generally show a good agreement with the ground truth data over the entire range of Dm. Overall, the KRR predictions are very close to the true values throughout the entire range of Dm for all nine cases in Fig. . The ANN predictions slightly deviate from the true value for cases with larger fcoating. For example, in the case of fcoating = 90 % and Df = 1.5, ANN underestimates the Q^abs. showed comparatively more deviation in the predictions for larger pure BC fractal particles than smaller particles. In this study, KRR and ANN predictions were consistently good for pure BC fractal particles (first row in Fig. ), although we could observe deviations from the true values for large and aged coated particle predictions (last row in Fig. ). Appendix  contains plots similar to Fig.  for the interpolation and extrapolation split. In general, errors increase with increasing aggregate sizes for the interpolation and extrapolation splits. The ML models we publish are based upon random-split experiments, and Fig.  shows how well both the ML methods provide accurate estimates of the optical properties of BC fractal aggregates at each aging stage.

Apart from making accurate predictions, our ML models should also be fast to provide a benefit over time-consuming simulations. Hence, we recorded the time needed to train on the entire training dataset and the time to make a single prediction in Table . As a result, the prediction time of both algorithms is less than 1 ms, which is a drastic improvement compared to the MSTM method, which can take up to 24 h, depending on the particle. It should be noted that the prediction time for ANN does not depend on the input data. Training the models takes comparatively longer, but it is usually done offline. Therefore, it is irrelevant for users using the pre-trained models we provide for their applications (see section “Code availability”).

Incorporating the fractal morphology of BC in global model calculations is essential, as the BC radiative forcing can increase up to 61 % compared to a more compact and aged particle . In the atmosphere, BC fractal aggregates are primarily found in conjunction with other aerosol types, such as organic carbon. It is therefore more relevant to predict the optical properties of BC fractal aggregates with organic coatings for atmospheric applications. To give an example of applying the ML algorithm to real-world atmospheric research, we predicted the optical properties of laboratory-generated soot for experiments described in Table 1 of our previous study .

The ML-based predictions were compared to the averages of each experimental case, represented by one data point in Fig. . The ML results correspond to KRR, the default algorithm used in the prediction script. The details of the laboratory experiments and instrumentations are given in Appendix . Figure a compares the single-scattering albedo (ω^ML) predicted by the ML algorithm with the measured ω from the laboratory experiment. The ω^ML predictions are in good agreement with the measured results for a range of forganics going up to 55 %. The uncertainty of nearly 10 % in the measured SSA is well represented within the 95 % confidence band of the ML-based predictions. On the contrary, Fig. b demonstrates that, if the conventionally used Mie core–shell theory is used, the predictions are overestimated by a large margin. The ML predictions of MAC are also compared to the measured MAC and the Mie-based predictions, whose results are given in Fig.  of Appendix D. The predictions MAC^ML were found to be less sensitive to the change in Dmob. Due to a lack of monodisperse mass measurements, comparing the predictions and measured values is not so straightforward. However, one can see that the discrepancies in the ML-based predictions of MAC are comparatively lower than the Mie-derived MAC values.

The sensitivity in the predicted MAC and SSA as a function of change in input parameters, such as the Dmob, Df, fcoating, and a, have been extensively discussed by and . The recommendations given by the above studies have been adapted for obtaining the results in Figs.  and  and are discussed in detail in Appendix . For future applications, it is recommended that ambient or laboratory datasets with a resolution of more than 30 min are used to minimize the interference of instrumental uncertainty due to noisy data. Similarly, for ambient or laboratory closure studies, it is recommended that the model output be compared with averaged optical observations.

Based on the success of the ML-based approach in predicting the optical properties of coated BC particles, it has great potential for future development to predict the optical properties of mixtures of BC and other aerosols. Because such a study would be exhaustive, we initially tested this approach on BC fractal aggregates and organic coatings to determine its effectiveness. Further research is necessary to develop an ML algorithm with features representing different morphological shapes and other chemical compositions, such as inorganics. In the long run, the goal should be to develop an ML algorithm that can be used to integrate all atmospheric aerosols into global climate models. To develop such a universal algorithm for all atmospheric aerosols, we must incorporate the conventional spherically shaped particles into the current prediction algorithm to represent the fraction of aged aerosols. In this study, due to the experimental design of , we could only test the ML-based prediction algorithm for particles with forganics of less than 65 %. The extension of the current algorithm to include more parameters also demands closure studies using more datasets of laboratory and ambient measurements.

The experiments conducted for this study show that our ML methods predict the optical properties of BC fractal aggregates with high accuracy as long as they are trained on sufficient data. However, the interpolation and extrapolation experiments show that the performance of both KRR and ANN significantly deteriorates when entirely removing certain ranges from the training data. This suggests that our models possess only limited generalization capabilities. Still, it should be noted that we train the models for practical use on the entire physically feasible range of Df and fcoating. Hence, those models will not have to extrapolate for any reasonable inputs.

Our models treat the wavelength λ as a continuous variable, meaning they should support computing optical properties at wavelengths that are not part of the training data. The prediction script can predict the optical properties well for the range between 467 and 660 and points close to the upper and lower limit. However, we did not test the models' generalization capabilities about the wavelength, since omitting just one wavelength from the training data would reduce the dataset size by one-third. Generating more ground truth data for other wavelengths requires refractive indices of BC and organics for that specific wavelength, which are unavailable in the literature. Even if they were available, it would be time-consuming, as MSTM simulations can take a long time to compute. Nevertheless, examining the models' generalization capabilities on other wavelengths in the future would be interesting.

In this study, the ML-based prediction algorithm is developed using training data of Npp up to 1000, which corresponded to particles with maximum Dmob of 1561 nm depending on the fcoating. This range of particle sizes was chosen while designing the database, considering the realistic size of BC-containing particles in the atmosphere. TEM analysis has shown a high probability that the BC-containing particles less than 1500 nm will be fractal . The ML algorithm developed in this study, which is based on a close-shell coating model, is suitable for such particles smaller than 1500 nm. However, when aerosol particles grow larger, the mass of BC decreases significantly compared to the mass of coating . For such cases of aged BC, using the conventional core–shell-based spherical morphology is appropriate. This is why we limited our training data range for particle size to 1561 nm. However, as demonstrated by , adding a few points in the training data significantly improves the extrapolation efficiency of machine learning models. Furthermore, some studies show that the optical properties are not sensitive to the change in the primary particle size a. Therefore, we fixed the ai to 15 nm and changed ao from 15.1 to 29 depending on the fcoating. Similarly to the parameters related to a particle size such as Npp, ro, and Dm, adding a few data points to the ai or ao can help optimize the extrapolation ability of the ML-based prediction algorithm. Although future studies can extend the model's extrapolation ability, the particle size range of the current prediction algorithm covers the physically feasible cases for BC fractal aggregates.

Both KRR and ANN provide only a single-point prediction for each input. In particular, their estimate does not quantify any uncertainty in the prediction. Bayesian ML methods such as Gaussian process regression can provide information about the uncertainty of a prediction via credible intervals as they return an entire probability distribution instead of a single-point estimate. Thus, it would be interesting to examine Bayesian ML for the prediction of BC fractal aggregates' optical properties. This method could be further developed for reporting the predictions for an ensemble of BC-containing aerosols with various physicochemical properties. However, applying them directly to our problem is not trivial, since the assumptions made by their statistical model (e.g., target variables follow a multivariate Gaussian distribution) are often violated in practice. Therefore, we leave the application of Bayesian ML to the BC aerosol problem to future work.

Atmospheric BC can exhibit a wide range of morphologies showing diversity at different locations . It was observed that aged transported soot can retain its fractal morphology 500 to 1000 km downwind of emission sources . The current state of the art for representing atmospheric soot particles focuses on spherical morphology . The model provided in this study was designed to simulate the optical properties for the entire BC life cycle, capturing the transition between fresh fractal and aged spherical particles. Furthermore, the calibration of light-absorption measurement devices is mostly done with fresh soot. We can link to atmospheric-relevant absorption by simulating mass absorption cross-sections and light-absorption enhancement factors. The coating model used in this study is called the “closed-cell model”, and the results showed good comparability with the realistic coating model . A more sophisticated coating model would be a good choice, but it requires more complex scattering models such as discrete dipole approximation (DDA), which is computationally expensive. With the DDA method, generating elaborate datasets for training ML algorithms is not feasible. We provide a method that predicts the optical properties of a wide range of ambient soot particles with high accuracy. Therefore, the results of this study are valuable for the simulation of realistic scenarios, despite the model limitations. There is scope for future studies to extend such an ML-based approach using other morphological models of BC and coating positions.

The present study demonstrated that the predictions of BC optical properties can be improved by incorporating their realistic morphologies. Unlike the computationally intensive simulations of complex scattering models, the ML-based approach accurately predicts optical properties in fractions of a second. In conjunction with a laboratory dataset, it was shown that optical properties like single-scattering albedo ω and mass absorption cross-section (MAC) can be predicted with greater accuracy than with a Mie-based approach. Using an extensive database for the physicochemical and optical properties of BC fractal aggregates, we trained two ML models – KRR and ANN – that can be used to predict the optical properties of coated BC aggregates at all aging stages. In particular, we could accurately predict the optical properties in the visible spectrum for BC fractal aggregates of any desired size, shape, and fraction of organic coating. Thus, this work illustrates the use of this realistic approach in real-world atmospheric research applications.

We summarize the key conclusions of the study as follows.

Active investigation area. BC is a highly relevant and active field of research, as it affects the climate system and human health. Global climate models require information about the optical properties of BC to simulate their radiative forcing. BC research will benefit from using this ML algorithm to generate the optical properties of BC based on more realistic fractal aggregates.

In summary, we demonstrated the feasibility of incorporating the realistic morphology of BC to improve the predictions of optical properties using a first-of-its-kind machine learning approach. This ML-based approach constitutes a significant step forward in BC aerosol research in two ways: firstly, it is the first attempt to provide optical properties of coated BC fractal aggregates at different stages of atmospheric aging using realistic representations. Secondly, this approach significantly reduces the heavy computational costs of using previous complex scattering models. Previous studies of BC avoid using complex scattering theories because of the high computational costs and prefer the more simplistic Mie theory. This research will be further developed in the future with the final goal of accurately predicting the optical properties of any mixture of atmospheric aerosols. We will investigate if the spherical core–shell model can be combined with the fractal aggregate-based ML model to distribute the weightage of light-absorption predictions for an ensemble of atmospheric BC aerosols with variable aging stages.