Validating a Model Predicting Retrieval Ordering Performance with Statistically Dependent Binary Features

The Information Retrieval Dependence (IRD) model predicts retrieval performance, with some or all dependencies and where there are binary features. Simulations with the Information Retrieval Validation (IRV) software are described that have been used to validate the IRD predictive model, showing that the IRD model accurately or exactly predicts retrieval performance under a variety of conditions. Instead of using traditional research methods using a sample of realistic documents and realistic queries, the authors exhaustively examine all document, query, and relevance combinations within a certain size range. While each of the numbers of components may be small, going through all the permutations of the relevance judgments, terms, and documents produced in one situation 551370 predictions, all of them matching the empirical ordering, suggesting that the predicting method is valid and accurate.