Analysis of Regularization in Deep Learning Models on Testbed Architectures

Abstract

Deep Learning models have come into significant use in the field of biology and healthcare, genomics, medical imaging, EEGs, and electronic medical records [1–4]. In the training these models can be affected due to overfitting, which is mainly due to the fact that Deep Learning models try to adapt as much as possible to the training data, looking for the decrease of the training error which leads to the increase of the validation error. To avoid this, different techniques have been developed to reduce overfitting, among which are the Lasso and Ridge regularization, weight decay, batch normalization, early stopping, data augmentation and dropout. In this research, the impact of the neural network architecture, the batch size and the value of the dropout on the decrease of overfitting, as well as on the time of execution of the tests, is analyzed. As identified in the tests, the neural network architectures with the highest number of hidden layers are the ones that try to adapt to the training data set, which makes them more prone to overfitting.