Decomposition of Discrete Choice Model Generated Probabilities and their Robustness to Changing Substantive Knowledge (Conditioning Variables)

Clear understanding of “goodness” and how substantive knowledge contributes to such goodness is generally absent from the economist’s use of probability. Although probability forecast from either subjective experts or from data based on prior theory and models can be generated, it is more problematic to generate a “good probability forecast” with a crisp understanding of what constitutes “good”. Further it is generally not clear to economists how different conditioning information affects this measure of “good.” Heretofore probability forecasts have been evaluated using the Brier Score and its Yates partition. Our work contributes by exploring how different sets of substantive information affect the Brier score and each component of the Yates partition. We will explore partitions associated with a set of observational data on beverages and the associated consumer decision to purchase. Probabilities are modeled using discrete choice models. Results show the higher the substantive knowledge, higher the model’s ability to offer a high probability for events occurred versus low probability for events that did not occur. Also, this model gave rise to lower Brier Score (lower the better) and higher covariance between probabilities offered and events observed. Better sorting of probabilities was demonstrated in the model with more substantive knowledge.