Handling class imbalance in customer churn prediction
Customer churn is often a rare event in service industries, but of great interest and great value. Until recently, however, class imbalance has not received much attention in the context of data mining (Weiss, 2004). In this study, we investigate how we can better handle class imbalance in churn prediction. Using more appropriate evaluation metrics (AUC, lift), we investigated the increase in performance of sampling (both random and advanced under-sampling) and two specific modelling techniques (gradient boosting and weighted random forests) compared to some standard modelling techniques. AUC and lift prove to be good evaluation metrics. AUC does not depend on a threshold, and is therefore a better overall evaluation metric compared to accuracy. Lift is very much related to accuracy, but has the advantage of being well used in marketing practice (Ling and Li, 1998). Results show that under-sampling can lead to improved prediction accuracy, especially when evaluated with AUC. Unlike Ling and Li (1998), we find that there is no need to under-sample so that there are as many churners in your training set as non churners. Results show no increase in predictive performance when using the advanced sampling technique CUBE in this study. This is in line with findings of Japkowicz (2000), who noted that using sophisticated sampling techniques did not give any clear advantage. Weighted random forests, as a cost-sensitive learner, performs significantly better compared to random forests, and is therefore advised. It should, however always be compared to logistic regression. Boosting is a very robust classifier, but never outperforms any other technique.
|Date of creation:||May 2008|
|Date of revision:|
|Contact details of provider:|| Postal: Hoveniersberg 4, B-9000 Gent|
Phone: ++ 32 (0) 9 264 34 61
Fax: ++ 32 (0) 9 264 35 92
Web page: http://www.ugent.be/eb
More information through EDIRC
Please report citation or reference errors to , or , if you are the registered author of the cited work, log in to your RePEc Author Service profile, click on "citations" and make appropriate adjustments.:
- A. Prinzie & D. Van Den Poel, 2007. "Random Forrests for Multiclass classification: Random Multinomial Logit," Working Papers of Faculty of Economics and Business Administration, Ghent University, Belgium 07/435, Ghent University, Faculty of Economics and Business Administration.
- Young-Hoon Park & Peter S. Fader, 2004. "Modeling Browsing Behavior at Multiple Websites," Marketing Science, INFORMS, vol. 23(3), pages 280-303, May.
- Baesens, Bart & Viaene, Stijn & Van den Poel, Dirk & Vanthienen, Jan & Dedene, Guido, 2002. "Bayesian neural network learning for repeat purchase modelling in direct marketing," European Journal of Operational Research, Elsevier, vol. 138(1), pages 191-211, April.
- G. Verstraeten & D. Van Den Poel, 2006. "Using Predicted Outcome Stratified Sampling to Reduce the Variability in Predictive Performance of a One-Shot Train-and-Test Split for Individual Customer Predictions," Working Papers of Faculty of Economics and Business Administration, Ghent University, Belgium 06/360, Ghent University, Faculty of Economics and Business Administration.
- Alan L. Montgomery & Shibo Li & Kannan Srinivasan & John C. Liechty, 2004. "Modeling Online Browsing and Path Analysis Using Clickstream Data," Marketing Science, INFORMS, vol. 23(4), pages 579-595, November.
- Joffre Swait & Rick L. Andrews, 2003. "Enriching Scanner Panel Models with Choice Experiments," Marketing Science, INFORMS, vol. 22(4), pages 442-460, September.
- B. Larivière & D. Van Den Poel, 2004. "Predicting Customer Retention and Profitability by Using Random Forests and Regression Forests Techniques," Working Papers of Faculty of Economics and Business Administration, Ghent University, Belgium 04/282, Ghent University, Faculty of Economics and Business Administration.
- B. Larivière & D. Van Den Poel, 2004. "Investigating the role of product features in preventing customer churn, by using survival analysis and choice modeling: The case of financial services," Working Papers of Faculty of Economics and Business Administration, Ghent University, Belgium 04/223, Ghent University, Faculty of Economics and Business Administration.
When requesting a correction, please mention this item's handle: RePEc:rug:rugwps:08/517. See general information about how to correct material in RePEc.
For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: (Nathalie Verhaeghe)
If references are entirely missing, you can add them using this form.