Challenges for the Repeatability of Deep Learning Models

Abstract

Deep learning training typically starts with a random sampling initialization approach to set the weights of trainable layers. Therefore, different and/or uncontrolled weight initialization prevents learning the same model multiple times. Consequently, such models yield different results during testing. However, even with the exact same initialization for the weights, a lack of repeatability, replicability, and reproducibility may still be observed during deep learning for many reasons such as software versions, implementation variations, and hardware differences. In this article, we study repeatability when training deep learning models for segmentation and classification tasks using U-Net and LeNet-5 architectures in two development environments Pytorch and Keras (with TensorFlow backend). We show that even with the available control of randomization in Keras and TensorFlow, there are uncontrolled randomizations. We also show repeatable results for the same deep learning architectures using the Pytorch deep learning library. Finally, we discuss variations in the implementation of the weight initialization algorithm across deep learning libraries as a source of uncontrolled error in deep learning results.