Realistic Data Delays and Alternative Inactivity Definitions in Telecom Churn: Investigating Concept Drift Using a Sliding-Window Approach

Abstract

Predicting customer churn is essential for telecommunications companies to maintain profitability. However, training models on historical models leads to performance degradation when they are applied to future conditions—a phenomenon known as concept drift. We employ a sliding-window approach that separates the training and testing time windows, creating a future-based “true test”. Using unique real data, we show that a CatBoost classifier model trained on older data can remain relevant when new, unseen intervals are used. A key innovation of our work is the use of 40-day “partial churn” labels; a model trained on these labels accurately predicts 90-day churn by simply adjusting the decision threshold. Out of the six modeled scenarios, in the main realistic scenario, CatBoost retained an accuracy above 0.798 and an F1 of near 0.704, reflecting its robustness even under real-world delays and potential drift. Overall, our findings emphasize that models do not necessarily “expire” with time; rather, their performance varies according to when they are tested. This research underscores the importance of a truly future-based evaluation (instead of artificial splits) and offers practical guidance for earlier churn detection when facing real-world data delays.