Relevance disagreements and unclear request forms

Disagreements about the relevance of documents to retrieval requests occur because relevance judges differently interpret requests or documents. Requests may be differently interpreted because they are unclear. Well‐known types of request obscurity are reviewed. Less well known is that a request may be unclear because its form—“documents about subject S,” “documents answering question Q,” etc.—is unclear. Explications are developed of the meanings of the request forms just given and several others. A request of any of the forms discussed is interpreted to be for documents which support statements of a specified kind in a specified way. For example, an “about S” request requires documents supporting statements which contain expression S (though several qualifications are needed); a “question Q” request requires documents which support answers to Q. Examples are given which suggest that some, perhaps all, “about S” requests are unclear. Some ways of formulating clear question requests are given. Various ways in which documents may support statements are distinguished. These depend on such factors as parts of a document used, inference strength, and background knowledge permitted. Some possibly clear support specifications are indicated.