Transfer learning with adaptive fine-tuning

Abstract

With the utilization of deep learning approaches, the key factors for a successful application are sufficient datasets with reliable ground truth, which are generally not easy to obtain, especially in the field of medicine. In recent years, this issue has been commonly addressed with the exploitation of transfer learning via fine-tuning, which enables us to start with a model, pre-trained for a specific task, and then fine-tune (train) only certain layers of the neural network for a related but different target task. However, the selection of fine-tunable layers is one of the major problems of such an approach. Since there is no general rule on how to select layers in order to achieve the highest possible performance, we developed the Differential Evolution based Fine-Tuning (DEFT) method for the selection of fine-tunable layers for a target dataset under the given constraints. The method was evaluated against the problem of identifying the osteosarcoma from the medical imaging dataset. The performance was compared against a conventionally trained convolutional neural network, a pre-trained model, and the model trained using a fine-tuning approach with manually handpicked fine-tunable layers. In terms of classification accuracy, our proposed method outperformed the compared methods by a margin of 4.45% to 32.75%.