Exploring Key XGBoost Hyperparameters: A Study on Optimal Search Spaces and Practical Recommendations for Regression and Classification

Abstract

This paper presents an in-depth exploration of four key hyperparameters—min_child_weight, n_estimators, max_depth, and learning_rate—in XGBoost (eXtreme Gradient Boosting), a machine learning algorithm recognized for its efficacy in diverse regression and classification tasks. By evaluating these hyperparameters across six datasets, we assess their impact on model performance in both regression and classification contexts. Given the computational demands of hyperparameter tuning, especially with large datasets, we propose an initial search space for each parameter to balance performance gains with resource efficiency. This approach offers practical guidance for optimizing XGBoost while mitigating the high computational costs associated with extensive searches. The findings provide insights into the role of each parameter in enhancing model accuracy and generalization, supporting practitioners in making effective tuning decisions for real-world applications.