Improving Model Accuracy with Probability Scoring Machine Learning Models

Abstract

Binary classification problems are exceedingly common across corporations, regardless of their industry, with examples including predicting attrition or classifying patients as high-risk vs low-risk. The motivation for this research is to determine techniques that improve prediction accuracy for operationalized models. Collaborating with a national partner, we conducted feature experiments to isolate industry-agnostic factors with the most significant impact on conversion rate. We also use probability scoring to highlight incremental changes in accuracy while we applied several improvement techniques to determine which would significantly increases a model’s predictive power. We compare five algorithms: XGBoost, LGBoost, CatBoost, and MLP, and an Ensemble. Our results highlight the superior accuracy of the ensemble, with a final log loss value of 0.5784. We also note that the highest levels of improvement in log loss occurs at the beginning of the process, after downsampling and using engineered custom metrics as inputs to the models