Using Predicted Outcome Stratified Sampling to Reduce the Variability in Predictive Performance of a One-Shot Train-and-Test Split for Individual Customer Predictions
AbstractSince it is generally recognized that models evaluated on the data that was used for constructing them are overly optimistic, in predictive modeling practice, the assessment of a model’s predictive performance frequently relies on a one-shot train-and-test split between observations used for estimating a model, and those used for validating it. Previous research has indicated the usefulness of stratified sampling for reducing the variation in predictive performance in a linear regression application. In this paper, we validate the previous findings on six real-life European predictive modeling applications for marketing and credit scoring using a dichotomous outcome variable. We find confirmation for the reduction in variability using a procedure we describe as predicted outcome stratified sampling in a logistic regression model, and we find that the gain in variation reduction is – also in large data sets – almost always significant, and in certain applications markedly high.
Download InfoIf you experience problems downloading a file, check if you have the proper application to view it first. In case of further problems read the IDEAS help page. Note that these files are not on the IDEAS site. Please be patient as the files may be large.
Bibliographic InfoPaper provided by Ghent University, Faculty of Economics and Business Administration in its series Working Papers of Faculty of Economics and Business Administration, Ghent University, Belgium with number 06/360.
Length: 10 pages
Date of creation: Jan 2006
Date of revision:
This paper has been announced in the following NEP Reports:
- NEP-ALL-2006-03-11 (All new papers)
- NEP-ECM-2006-03-11 (Econometrics)
- NEP-MKT-2006-03-11 (Marketing)
Please report citation or reference errors to , or , if you are the registered author of the cited work, log in to your RePEc Author Service profile, click on "citations" and make appropriate adjustments.:
- Dasgupta, Chanda Ghose & Dispensa, Gary S. & Ghose, Sanjoy, 1994. "Comparing the predictive performance of a neural network model with some traditional market response models," International Journal of Forecasting, Elsevier, vol. 10(2), pages 235-244, September.
- Alan L. Montgomery & Shibo Li & Kannan Srinivasan & John C. Liechty, 2004. "Modeling Online Browsing and Path Analysis Using Clickstream Data," Marketing Science, INFORMS, vol. 23(4), pages 579-595, November.
- Joffre Swait & Rick L. Andrews, 2003. "Enriching Scanner Panel Models with Choice Experiments," Marketing Science, INFORMS, vol. 22(4), pages 442-460, September.
- J. Burez & D. Van Den Poel, 2008. "Handling class imbalance in customer churn prediction," Working Papers of Faculty of Economics and Business Administration, Ghent University, Belgium 08/517, Ghent University, Faculty of Economics and Business Administration.
For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: (Nathalie Verhaeghe).
If references are entirely missing, you can add them using this form.