XTOOS: Stata module for evaluating the out-of-sample prediction performance of panel-data models

The package XTOOS includes four new commands that allow to evaluate the out-of-sample prediction performance of panel-data models in their time-series and cross-individual dimensions separately, also with separate procedures for different types of dependent variables—either continuous or dichotomous variables. The name of the commands are xtoos_t, xtoos_i, xtoos_bin_t, and xtoos_bin_i. The time-series procedures (xtoos_t and xtoos_bin_t) exclude a number of time periods defined by the user from the estimation sample for each individual in the panel. Similarly, the cross-individual procedures (xtoos_i and xtoos_bin_i) exclude a group of individuals (for example, countries) defined by the user from the estimation sample (including all their observations throughout time). Then, for the remaining subsamples (training-sample), they fit the specified models and use the resulting parameters to forecast the dependent variable (or the probability of a positive outcome) in the unused time-periods or the unused individuals (testing-sample). The unused time-period or individual sets are then recursively reduced by one period in every subsequent step in the time-series dimension, or in either a random or an ordered fashion in the cross-individual dimension. The estimation and forecasting evaluation are repeated until there are no more periods ahead or more individuals that could be evaluated. In the continuous cases, the model's forecasting performance is reported both in absolute terms (RMSE) and relative to a naive prediction, by means of a U-Theil ratio. In the dichotomous case, the prediction performance is evaluated based on the area under the receiver operator characteristic (ROC) statistic evaluated in both the training sample and the out of sample. The procedures allow to choose different estimation methods, including some dynamic methodologies, and could also be used in a time-series or a cross-section dataset only. They also allow evaluating the model's forecasting performance for one particular individual or for a defined group of individuals instead of the whole panel.