These authors contributed equally to this work.

It is necessary to accurately determine the optical properties of highly absorbing black carbon (BC) aerosols to estimate their climate impact. In the past, there has been hesitation about using realistic fractal morphologies when simulating BC optical properties due to the complexity involved in the simulations and the cost of the computations. In this work, we demonstrate that, by using a benchmark machine learning (ML) algorithm, it is possible to make fast and highly accurate predictions of the optical properties for BC fractal aggregates. The mean absolute errors (MAEs) for the optical efficiencies ranged between 0.002 and 0.004, whereas they ranged between 0.003 and 0.004 for the asymmetry parameter. Unlike the computationally intensive simulations of complex scattering models, the ML-based approach accurately predicts optical properties in a fraction of a second. Physiochemical properties of BC, such as total particle size (number of primary particles (

Black carbon (BC) aerosols are strong absorbers of solar radiation formed from incomplete combustion of fossil fuels, biofuels, and biomass

High-resolution transmission electron microscopy (TEM) images showed that the BC particles have a fractal structure composed of numerous spherules known as primary particles

Consequently, pre-calculated databases have been developed for aggregate properties to save time in constructing detailed aggregates and time-consuming optical simulations

This study demonstrates the use of a machine-learning-based approach to predict the optical properties of BC aggregates at various aging stages, including coating, which is highly relevant for atmospheric aerosols. Combining this ML-based approach with a laboratory dataset showed that optical properties like single-scattering albedo (

The paper is structured as follows: Sect. 2 provides an overview of the physical, chemical, and optical properties of BC used in this study. Section 3 describes the machine learning techniques, including the data processing, machine learning algorithms, and evaluation procedures. In Sect. 4, the results demonstrate that realistic morphologies of BC can be used to accurately predict optical properties at various stages of aging. Section 5 discusses how the results compare to laboratory measurements of BC, discussing the atmospheric processing in detail. Potential limitations and challenges of this work are presented in Sect. 6, and we end with the main conclusions in Sect. 7.

The database for the physicochemical and optical properties of BC fractal aggregates has been designed to consider all the possible aging stages of BC. The optical properties of BC fractal aggregates are most sensitive to the change in particle size as they age

Overview of the various features of the database for physicochemical and optical properties of black carbon fractal aggregates. The features are arranged based on the three steps of constructing this database. As the legend at the bottom indicates, the features are further divided into physicochemical properties, optical properties, and others.

The BC fractal aggregate's physicochemical features include size, mass, volume, morphology, and composition. Figure

Visualization of the various BC aggregate particles generated in this study. Fresh BC aggregates with no external coating are shown in panels

Along with BC, a complex mixture of gas-phase organic compounds is co-emitted during incomplete combustion, forming a coating around the BC aggregates

The tunable diffusion-limited aggregation (DLA) software

The real (

As mentioned in Sect.

The subset of the database used as input was designed to include the critical parameters that influence the BC optical properties. As mentioned in Sect.

Similarly, a BC fractal aggregate's optical properties are also not independent. Thus, we make the ML model predict only the following three properties and compute the rest using the formulae in Sect.

After feature selection, we transform input features using the Box–Cox transformation

Given a labeled dataset of

A popular choice for the kernel function is the Gaussian or radial basis function (RBF) kernel

We use scikit-learn's KRR implementation

Artificial neural networks (ANNs) constitute one of the founding pillars of ML's success during the last 10 years. Originally, their design was inspired by the structure of neurons inside the nervous system of several organisms

In our experiments, we use a feed-forward ANN, sometimes also called a multi-layer perceptron (MLP). It consists of an arbitrary number (

Formally, we can define an MLP as a function

The number of hidden layers, the number of neurons in those hidden layers, and the activation function are usually chosen by a human before training a neural network. Together, they define the architecture of the MLP. We can learn values for the parameters

Note that, in general, Eq. (

For our experiments, we implemented an MLP using Keras

In the case of kernel ridge regression, regularization is carried out by the regularization constant

We use the mean absolute error (MAE) as our primary performance metric: given a dataset

Regardless of the split strategy, we split the training set once more into a train and a validation set using the random-split method during the training phase. Here, we again use 30 % of the data for validation and the remaining 70 % for training. Our models then train on the train set for all possible hyperparameter configurations defined in the grid, and we record the MAE on the validation set for each combination. Finally, we choose the combination with the lowest MAE and evaluate the corresponding model's MAE on the test set.

The error distributions for the ML methods are presented in Fig.

Boxplots summarizing the error between the predicted value (

The MAEs for our experiments are reported in Table

Mean absolute errors of the predicted optical properties for different experiments. The training data for the interpolation split consist of fractal dimensions in

A one-to-one comparison was performed between the estimates and true values to understand better how the ML methods predict optical properties. Figure

Comparison of the predicted optical properties with their true values when the ML models are trained on a random subset of data. The data points for predicted optical properties correspond to KRR and ANN, as shown by the legend on the top right. The blue line in each panel of the figure corresponds to the one-to-one line between the

During their lifetime, BC fractal aggregates undergo complex changes in size, composition, and morphology due to atmospheric processing. Figure

Absorption efficiency (

Apart from making accurate predictions, our ML models should also be fast to provide a benefit over time-consuming simulations. Hence, we recorded the time needed to train on the entire training dataset and the time to make a single prediction in Table

Training time for 18 526 samples in the dataset and prediction time per sample in seconds. Values were recorded on a machine with Intel(R) Core(TM) i7-9750H CPU, 8 GB RAM, and NVIDIA GeForce GTX 1650 GPU.

Incorporating the fractal morphology of BC in global model calculations is essential, as the BC radiative forcing can increase up to 61 % compared to a more compact and aged particle

The ML-based predictions were compared to the averages of each experimental case, represented by one data point in Fig.

The sensitivity in the predicted MAC and SSA as a function of change in input parameters, such as the

Single-scattering albedo

Based on the success of the ML-based approach in predicting the optical properties of coated BC particles, it has great potential for future development to predict the optical properties of mixtures of BC and other aerosols. Because such a study would be exhaustive, we initially tested this approach on BC fractal aggregates and organic coatings to determine its effectiveness. Further research is necessary to develop an ML algorithm with features representing different morphological shapes and other chemical compositions, such as inorganics. In the long run, the goal should be to develop an ML algorithm that can be used to integrate all atmospheric aerosols into global climate models. To develop such a universal algorithm for all atmospheric aerosols, we must incorporate the conventional spherically shaped particles into the current prediction algorithm to represent the fraction of aged aerosols. In this study, due to the experimental design of

The experiments conducted for this study show that our ML methods predict the optical properties of BC fractal aggregates with high accuracy as long as they are trained on sufficient data. However, the interpolation and extrapolation experiments show that the performance of both KRR and ANN significantly deteriorates when entirely removing certain ranges from the training data. This suggests that our models possess only limited generalization capabilities. Still, it should be noted that we train the models for practical use on the entire physically feasible range of

Our models treat the wavelength

In this study, the ML-based prediction algorithm is developed using training data of

Both KRR and ANN provide only a single-point prediction for each input. In particular, their estimate does not quantify any uncertainty in the prediction. Bayesian ML methods such as Gaussian process regression

Atmospheric BC can exhibit a wide range of morphologies showing diversity at different locations

The present study demonstrated that the predictions of BC optical properties can be improved by incorporating their realistic morphologies. Unlike the computationally intensive simulations of complex scattering models, the ML-based approach accurately predicts optical properties in fractions of a second. In conjunction with a laboratory dataset, it was shown that optical properties like single-scattering albedo

We summarize the key conclusions of the study as follows.

In summary, we demonstrated the feasibility of incorporating the realistic morphology of BC to improve the predictions of optical properties using a first-of-its-kind machine learning approach. This ML-based approach constitutes a significant step forward in BC aerosol research in two ways: firstly, it is the first attempt to provide optical properties of coated BC fractal aggregates at different stages of atmospheric aging using realistic representations. Secondly, this approach significantly reduces the heavy computational costs of using previous complex scattering models. Previous studies of BC avoid using complex scattering theories because of the high computational costs and prefer the more simplistic Mie theory. This research will be further developed in the future with the final goal of accurately predicting the optical properties of any mixture of atmospheric aerosols. We will investigate if the spherical core–shell model can be combined with the fractal aggregate-based ML model to distribute the weightage of light-absorption predictions for an ensemble of atmospheric BC aerosols with variable aging stages.

The volume equivalent radius (

The mobility diameter of a sphere (

The relationship between the outer radius of the primary particle (

The optical cross-sections (

The single-scattering albedo (

The total mass absorption cross-section (

Features from the database of physicochemical and optical properties of black carbon fractal aggregates. For independent features, the list of values is provided. The features for which the range has provided correspond to dependent features.

Refractive indices (both real and imaginary parts) of BC and organics at various wavelengths in the visible range

Previous machine learning studies.

Hyperparameter values for the kernel ridge regression (KRR) experiments along with the optimal value for each parameter.

Hyperparameter values for the multi-layer perceptron (MLP) experiments along with the optimal value for each parameter.

Training range and test range of the features during the interpolation split.

Training range and test range of the features during the extrapolation split.

Maximum errors of different splits for their test sets.

Error between the predicted and true values for three optical properties. The residuals are shown when models are trained on data with different ranges of fractions of coating (

Figure

Error between the predicted value (

Comparison of the predicted optical properties with their true values for the interpolation split when the ML models are trained on data with boundary fractal dimensions (

Figures

Comparison of the predicted optical properties with their true values for extrapolation split when the ML models are trained on data with smaller fractal dimensions (

Figure

Optical properties of BC fractal aggregates predicted using machine learning methods KRR and ANN for the interpolation split when models are trained on data with boundary fractal dimensions (

Similarly, Fig.

Optical properties of BC fractal aggregates predicted using machine learning methods KRR and ANN for the extrapolation split when models are trained on data with smaller fractal dimensions (

The data from the laboratory experiments by

The input parameters used while running the prediction script are

Mass absorption cross-section (MAC) for coated BC particles generated in a laboratory study at different

The output parameters compared to the observations were SSA and MAC. The observational SSA was calculated from the ratio of

A Python script that predicts the optical properties of BC fractal aggregates using the trained ML-based models is available in a GitHub repository at

The dataset of simulated physiochemical and optical properties that we describe in Sect.

The supplement related to this article is available online at:

The study was designed by BR, ThM, JP, ToM, MP, and MK. BR and ThM developed the optical simulations and database. The machine learning experiments were conducted by JP and ToM, with help from BR and ThM. The results were prepared by JP and ToM, with help from BR. The paper was written by BR, JP, and ToM. The paper was reviewed, commented on, and edited by ThM, MK, and MP.

The contact author has declared that none of the authors has any competing interests.

Publisher’s note: Copernicus Publications remains neutral with regard to jurisdictional claims made in the text, published maps, institutional affiliations, or any other geographical representation in this paper. While Copernicus Publications makes every effort to include appropriate place names, the final responsibility lies with the authors.

We would like to thank the members of the EMPIR 16ENV02 Black Carbon project for their support and feedback. Marius Kloft acknowledges support by the Carl Zeiss Foundation; the DFG awards KL 2698/2-1, KL 2698/5-1, KL 2698/6-1, and KL 2698/7-1; and the BMBF awards 03|B0770E and 01|S21010C.

This research has been supported by the European Metrology Programme for Innovation and Research (EMPIR; grant no. 16ENV02 Black Carbon).

This paper was edited by Joshua Fu and reviewed by two anonymous referees.