Cyclical Learning Rate Optimization on Deep Learning Model for Brain Tumor Segmentation

Abstract

In recent years, deep learning has found widespread applications in tasks such as segmentation and classification. Fine-tuning hyperparameters is crucial to improve performance, with the learning rate being a key parameter. Various methods, including adaptive learning rates, learning rate scheduling, and cyclical learning rates, have been used to optimize learning rates. Cyclical learning rates offer significant benefits with minimal computational cost, as seen in previous research. This study introduces a novel approach to tuning the cyclical learning rate, which incorporates the exponential moving average. These methods are applied to the BraTS 2021 dataset for segmentation tasks, resulting in superior performance compared to the previous approach. Our proposed method reduces the epochs required to reach convergence by 19 and 54 epochs for U-Net and Dense U-net, respectively. For Res U-net, the epoch needed to convergence is 10 epochs more. However, the proposed method produces lower loss values with 0.707, 0.657, and 0.665 compared to the previous method with 0.712, 0.685, and 0.725 for U-net, Res U-net, and Dense U-net, respectively.