Improvement of climate predictions and reduction of their uncertainties using learning algorithms

Strobach, E.; Bel, G.

doi:https://doi.org/10.5194/acp-15-8631-2015

Articles | Volume 15, issue 15

https://doi.org/10.5194/acp-15-8631-2015

© Author(s) 2015. This work is distributed under
the Creative Commons Attribution 3.0 License.

https://doi.org/10.5194/acp-15-8631-2015

© Author(s) 2015. This work is distributed under
the Creative Commons Attribution 3.0 License.

Articles | Volume 15, issue 15

Research article

|

04 Aug 2015

Research article |

| 04 Aug 2015

Improvement of climate predictions and reduction of their uncertainties using learning algorithms

E. Strobach and G. Bel

Download

Final revised paper (published on 04 Aug 2015)
Preprint (discussion started on 12 Mar 2015)

Interactive discussion

Status: closed

AC: Author comment | RC: Referee comment | SC: Short comment | EC: Editor comment

- Printer-friendly version

- Supplement

RC C2381: 'Review of manuscript acp-2015-111', Anonymous Referee #1, 08 May 2015
- RC C2388: 'Correction to my posted review', Anonymous Referee #1, 08 May 2015
  - RC C2483: 'Clarification', Anonymous Referee #1, 12 May 2015
- AC C2967: 'Response to reviewers' comments', Golan Bel, 26 May 2015
RC C2501: 'Review of acp-2015-111', Anonymous Referee #2, 13 May 2015
- AC C2969: 'Response to reviewers' comments', Golan Bel, 26 May 2015

Peer-review completion

AR: Author's response | RR: Referee report | ED: Editor decision

AR by Golan Bel on behalf of the Authors (26 May 2015) Author's response

ED: Referee Nomination & Report Request started (01 Jun 2015) by V. Faye McNeill

RR by Anonymous Referee #2 (08 Jun 2015)

RR by Anonymous Referee #1 (04 Jul 2015)

Suggestions for revision or reasons for rejection

I commend the authors for their significant additional work and revisions. I vote for acceptance subject to technical corrections and minor revisions to address the following primary concerns. Note: the technical corrections may involve redoing their experiments to address item 1.

1. Parameter tuning: it is not clear whether the parameter tuning was done in a principled way.

2. Exposition needs to be revised to be more nuanced, so that it adequately reflects the observed results. Currently there are various assertions meant to explain the performance of the algorithms, that are unsupported. It would be more scientifically accurate to discuss the results without asserting such strong conclusions.

1. Parameter tuning

The authors state that the parameter eta (learning rate) used in EGA and EWA was "was set to achieve the best performance during the learning period." It is unclear whether the parameter tuning was done in a principled way. Parameter tuning needs to be done on a separate subset of the data set that is not part of the training set or test set. Alternatively, there are also techniques using cross-validation or bootstrap. The authors need be explicit about what they did, and provide details such as how many parameter values were tested, which values were ultimately chosen, and how they formed the validation data set. Also, was the alpha-initialization procedure for LAA done using the regret-optimal technique from MJ03?

For an introduction on how to tune parameters in machine learning algorithms, that addresses the need for a separate "validation set" that is not part of the training set, see for example p. 222 (and all of Chapter 7 on Model Assessment and Selection) of:
Trevor Hastie, Robert Tibshirani, Jerome Friedman: The Elements of Statistical Learning: Data Mining, Inference, and Prediction. Springer. 2009.

These issues need to be addressed in order to make sure that a fair comparison is being made between the various algorithms. This is important since the authors draw conclusions from these comparisons, such as that non-stationarity was not an issue. (Note: this is quite a strong assertion; point 2. advises toning down the exposition to avoid unsupported or weakly supported assertions.) The authors should clarify how they did parameter tuning, and may actually need to re-run the parameter tuning for the algorithms with parameters, and then update their results on the test set.

2. Expository revisions needed to reduce unsubstantiated assertions

"As a successor of the EWA, the LAA also tends to converge to the best model.”
This is inaccurate; the fixed-share algorithm is explicitly designed to allow switches between models, and thus need not converge to one model (convergence will depend on the level of non-stationarity in the sequence of expert prediction losses, along with the value of the alpha parameter). Neither does Learn-alpha, so if it happened to converge to one model in their experiments, evidence should be provided.

Earlier in the paper, the following should be revised, to avoid the confusion above:
"The fixed-share algorithm is a generalization of the EWA algorithm that increases the ability to switch between experts"
The terms "generalization" and "increases the ability" are misleading — the fixed-share algorithm explicitly models switches between experts, whereas EWA does not.

Currently such claims appear multiple places in the paper, and at times are asserted as the explanation for certain results, e.g.
"Note that the smaller uncertainty of the EWA and the LAA forecasters is simply due to the fact that these forecasters converge to the best model in each grid cell (if the ensemble includes a model that is always the best)."
Not only is this statement inaccurate for LAA, but also it is not scientifically sound to assert a sole cause for an outcome, without significant evidence.

Along these lines, this strong assertion is also unsupported:
"The small improvement achieved by the EWA and LAA, when the climatology was added as an expert to the ensemble, is only associated with the fact that they tracked the climatology in regions where it performed the best."

It seems that statistical significance was studied for EGA only, and the differences between the performance of the 3 algorithms was not tested for statistical significance. If they were tested this should be clarified; currently the following statement seems like an assertion.
"The improvement of the LAA over the EWA is too small to be considered significant."

As mentioned above, given these issues and the reduction of uncertainty by LAA, the strong assertion of lack of non-stationarity does not seem sufficiently supported. Either further evidence should be provided, e.g. reporting statistics at a higher granularity than global, or else this can be addressed by modifying the exposition.

Minor comments:

- "optimal discretization" could sound misleading — this is the discretization that optimizes the performance guarantees in MJ03.
- It might be helpful to use the term "online learning," to introduce these types of algorithms to this audience. “SLA" is non-standard in machine learning.
- Did you find the result for Figure 4 to hold for LAA? With modeling non-stationarity, this might not hold.

Hide

ED: Reconsider after minor revisions (Editor review) (13 Jul 2015) by V. Faye McNeill

AR by Golan Bel on behalf of the Authors (19 Jul 2015) Author's response Manuscript

ED: Publish as is (20 Jul 2015) by V. Faye McNeill