How to best apply deep neural networks in geosciences: Towards optimal "averaging" in dropout training

Abstract

The main objectives of geosciences is to find the current state of the Earth-i.e., solve the corresponding inverse problems-and to use this knowledge for predicting the future events, such as earthquakes and volcanic eruptions. In both inverse and prediction problems, often, machine learning techniques are very efficient, and at present, the most efficient machine learning technique is deep neural training. To speed up this training, the current deep learning algorithms use dropout techniques: they train several sub-networks on different portions of data, and then "average" the results. A natural idea is to use arithmetic mean for this "averaging", but empirically, geometric mean works much better. In this paper, we provide a theoretical explanation for the empirical efficiency of selecting geometric mean as the "averaging" in dropout training.