Deep Learning for Multi-Output Regression Using Gradient Boosting

Abstract

This paper presents a novel methodology to address multi-output regression problems through the incorporation of deep-neural networks and gradient boosting. The proposed approach involves the use of dense layers as additive models within the Gradient Boosting framework using an auto transfer learning technique. At each boosting iteration, the deep model is cloned with the already trained layers frozen, and a new dense layer is concatenated to the frozen ones. Subsequently, only the weights of the newly added layer are trained in order to reduce the complexity of the learning task. Each layer is trained on the residuals of the squared loss function from previous iterations, resulting in the creation of a robust sequentially deep-trained neural network ensemble. Our experimental results demonstrate that the proposed approach leads to a significant improvement in the performance of the deep framework, resulting in more accurate predictions and improved model interpretability.