Addressing the Coupled Optimization of Feature Selection and Hyperparameter Tuning Using a TPE-Driven XGBoost-RFE Framework

Abstract

This study presents a methodological advancement for machine learning by developing a framework that solves the coupled problem of feature selection and hyperparameter optimization. The proposed TPE-XGBoost-RFE algorithm integrates a sequential model-based optimization technique, the Tree-structured Parzen Estimator (TPE), with a wrapper feature selection method. This approach concurrently searches for a globally optimal combination of predictive features and model hyperparameters. The efficacy of the framework is demonstrated in the task of predicting long-term tropospheric ozone concentrations. This integrated process identifies an optimal 22-feature subset, reducing dimensionality by 37% while simultaneously tuning nine key XGBoost hyperparameters. The robustness of this subset is validated across multiple machine learning models, all exhibiting superior predictive performance with lower error metrics compared to those trained on the full feature set or through a simpler filter-based method. This study demonstrates that a unified optimization strategy is critical for developing high-performing predictive models.