A Comparison Between Adam and Levenberg–Marquardt Optimizers for the Prediction of Extremes: Case Study for Flood Prediction with Artificial Neural Networks †

Abstract

Artificial neural networks (ANNs) adjust to the underlying behavior in the dataset using a training rule or optimizer. The most popular first-and second-order optimizers, Adam (AD) and Levenberg–Marquardt (LM), were compared with the aim of predicting extreme flash floods of a runoff-dominated hydrological system. A fully connected multilayer perceptron with a shallow structure was used to reduce complexity and limit overfitting. The inputs of the ANN were determined by rainfall–water level cross-correlation analysis. For each optimizer, the hyperparameters of the ANN were selected using a grid search and the cross-validation score on a novel criterion (PERS PEAK) mixing the persistency (PERS) and the quality of flood-peak restitution (PEAK). For an extreme and unseen event used as a test set, LM outperformed AD by 25% on all performance criteria. The peak water level of this event, 66% greater than that of the training set, was predicted by 92% after more training iterations were done by the LM optimizer. This shows that the ANN can predict beyond the ranges of the training set, given the right optimizer. Nevertheless, the LM training time was up to five times longer than that of AD during grid search.