A Fisher’s Exact Test Justification of the TF–IDF Term-Weighting Scheme

Term frequency–inverse document frequency, or TF–IDF for short, is arguably the most celebrated mathematical expression in the history of information retrieval. Conceived as a simple heuristic quantifying the extent to which a given term’s occurrences are concentrated in any one given document out of many, TF–IDF and its many variants are routinely used as term-weighting schemes in diverse text analysis applications. There is a growing body of scholarship dedicated to placing TF–IDF on a sound theoretical foundation. Building on that tradition, this article justifies the use of TF–IDF to the statistics community by demonstrating how the famed expression can be understood from a significance testing perspective. We show that the common TF–IDF variant TF–ICF is, under mild regularity conditions, closely related to the negative logarithm of the p-value from a one-tailed version of Fisher’s exact test of statistical significance. As a corollary, we establish a connection between TF–IDF and the said negative log-transformed p-value under certain idealized assumptions. We further demonstrate, as a limiting case, that this same quantity converges to TF–IDF in the limit of an infinitely large document collection. The Fisher’s exact test justification of TF–IDF equips the working statistician with a ready explanation of the term-weighting scheme’s long-established effectiveness. Supplementary materials for this article are available online.