Performance measures for information retrieval systems—an experimental approach

This study is an investigation into some critical factors which would constitute reliable performance measures for bibliographic information retrieval systems. Thirty‐two (32) judges, grouped by type (researcher or student), level (senior or junior), specialty (biomedicine or social science), and evaluation context (relevance or utility), were asked to rate 16 documents on alcohol studies which included four different search topics and four document citations and abstracts for each search topic. Half of the judges were asked to rate the documents on how relevant the documents were to the search topic; the other half rated the same documents on the basis of the document's perceived utility for the individual judge. The topics and documents were presented in a controlled prescribed order, so as to test the effects of learning on the evaluation process. After rating the documents, judges were also asked to rate the importance of five document attributes (author, title, abstract, source of publication, and date of publication), and six information attributes (accuracy, completeness, subject, suggestiveness, timeliness, and treatment). The study finds that there is no operational difference between the relevance‐theoretic and the utility‐theoretic model of evaluation. It further suggests the need for performance measures based upon a complex set of factors including document and information attributes, the judge, and other environmental factors, particularly the effects of learning during the evaluation process. © 1988 John Wiley & Sons, Inc.