Enhanced Logistic Regression Using Stacking Algorithm for Imbalanced and High-Dimensional Data

Abstract

In binary classification tasks, logistic regression models often perform poorly and can even fail when dealing with issues such as class imbalance and the curse of dimensionality. To address these problems, this paper proposes an improved logistic regression model based on the stacking approach. First, the method constructs multiple logistic regression sub-models by employing a dual randomization strategy on both samples and features. The specific strategy involves retaining a small number of minority class samples and drawing a corresponding number of samples from the majority class in proportion, while also randomly selecting features to form subsets. This strategy not only effectively alleviates the curse of dimensionality but also can handle the issue of sample class imbalance. Second, the method uses the probability outputs of the logistic regression sub-models as new predictive variables, and these encoded representations are used to train a new ensemble model, thereby enhancing the model's ability to balance fitting bias and prediction variance. Moreover, the classifier used to construct the new ensemble classification model is model-agnostic. Through validation with simulation experiments and real data analysis, the results show that this method significantly improves predictive performance compared to the original logistic regression model. Additionally, compared to some classical classification models, the proposed method also demonstrates strong competitiveness.