Some independent agreements and resolved disagreements about answer‐providing documents

Eighty‐two documents and 30 questions, in documentation and related areas, were compared to find answer‐providing documents (documents from which answers to questions can be inferred). Two judges (documentation experts) made comparisons independently. They discussed their disagreements, attempting to resolve them. In each case the positive judge (who had independently judged a document answer‐providing) was first asked to indicate what answer he inferred, and from what document passage(s). The further discussion depended on the details of each case. There were 32 independent agreements on positive judgments. There were 48 disagreements between independent judgments, all resolved by discussion. Thirty‐four resolutions were agreements on positive judgments, accomplished by pointing out overlooked passages, unnoticed connections, or alternative meanings. Fourteen resolutions were agreements on negative judgments, accomplished by pointing out document misinterpretations, the challenged positive judge being unable to describe an inference and joint work not finding one, or agreement that both judges lacked sufficient background knowledge. In general, the resolution procedures used will resolve a disagreement about whether a document is answerproviding or reduce it to a familiar kind of scientific disagreement (about a passage's meaning, a statement's correctness, or an inference's correctness). This seems better than treating relevance judgements as subjective and not open to rational discussion.