Truth set size prediction by Newton’s cooling law

Precision and recall are useful indices to evaluate an operation, algorithm, system, and other subjects from two different facets. However, they are not readily available when the subject is still in progress because the truth set, which is required to calculate recall, is unknown. In this study, a method to predict the size of the truth set of an inquiry still in progress is presented, which consists of a classical 18th century mechanics found and formulated by Isaac Newton, today known as “Newton’s cooling law”, with some set-theoretical trick and executed by Markov Chain Monte Carlo. The developed method is applied to nation-wide scale collections of identifications of the authors of academic articles as the affiliation data of Japanese national research organizations, and obtain recall values, as a part of objective, evidence-based policy for science and technology of the Japanese government. The author identification result is naturally represented as a bipartite directed graph, from the set of authors to the set of affiliation data. We conduct a sort of network prediction, not on the bipartite graph itself but on its vertices size and obtain the true graph size by using a simple and straightforward probabilistic model, which is implemented by also a classical, yet recently developing probabilistic inference method.Author summary: In this study, we propose a method to predict the value of recall of the unfinished work, for example, survey or inquiry still in progress. It may sound strange to discuss recall or precision on a subject still going on, yet, they are needed in the real world applications because we want to know how much more we can expect from the unfinished inquiry. The trick is to predict the size of the truth set, not the set itself. If we observe identical subject multiple times, the observations must be slightly different from each other, depending on how much left to be done. If the results are almost equal, the observations are good approximation of the truth set, and the recall should be high. Conversely, disagreement between the observations is a sign of low recall. The method consists of simple and classic early eighteenth century mechanics found and formulated by Sir Isaac Newton and known as “Newton’s cooling law” today, some mathematical tricks for data preparation, and Markov Chain Monte Carlo. We apply the method to nation-wide scale collections of article authors identification as the affiliation data of Japanese national research organizations, and predict how many researchers are left to be identified as authors. As the identification relation from an author to the affiliation data of a researcher is naturally represented as a directed bipartite graph, a form of network-prediction on a directed bipartite graph is executed in this study.