Retrieval language of social sciences and natural sciences: A statistical investigation

Current guidelines for retrieval thesaurus design and maintenance recommend the use of relative frequency of occurrence of descriptors for the design and updating of a thesaurus. This study examined the extent to which Zipf and Whatmough's theory of quantitative semantics provides the theoretical justification for the current practice. The retrieval language of social sciences and science/technology was analyzed to test if the observed data on relative frequency conforms to the theoretical distribution of Zipf's law. Findings demonstrate that the principle of least effort serves to explain the pattern of use of the retrieval language in social sciences but not in science/technology. For social sciences, one may assume that the frequently occurring descriptors are indeed becoming semantically ambiguous thus requiring a greater variety of access points to the concepts that the more frequently occurring descriptors represent. No theoretical justification was found for using the frequency data in the design and updating of the descriptors of science/technology. Thus, this finding challenges the theoretical basis for the present practice of using frequency data for the design and updating of the retrieval thesauri of science/technology.