Optimizing LightGBM for Regression: A Study on Parameter Influence and Performance

Abstract

The accurate forecasting of demand is a major challenge for production companies, especially for companies that engage in make-to-order production. Accurately anticipating demand enables companies to develop a robust production program and mitigate overloading or underutilization of production resources. Light Gradient Boosting Machine (LightGBM) is a Machine Learning (ML) algorithm, developed by Microsoft Research, capable of performing regression, classification, and ranking tasks. The algorithm gained attention in recent years due to its ability to efficiently process large datasets and a high number of features while still being cheap in terms of computational costs. However, the quality of the results largely depends on setting the right parameters carefully. Hyperparameter optimization, such as GridSearch, can be used to find suitable parameters, but these methods are very time-consuming and require users to limit both the number of parameters and the range of the respective values. This paper aims to research the impact of the parameters of LightGBM on the predictive performance of regression tasks. To achieve this task, a total of 2,592 simulated sales datasets were created, each varying in seasonality, seasonal duration, seasonal amplitude, linear growth and random noise. For each of the datasets a LightGBM model was trained, using hyperparameter optimization. The models were then compared using the Root Mean Squared Error (RMSE) as a metric to find the best performing models. A thorough analysis of these parameters provides insight into the importance of different parameters for regression tasks and can be utilized to speed up hyperparameter optimization of future regression LightGBM-based regression models.