Quantifying perceived impact of scientific publications

We report on an empirical verification of the degree to which citation numbers represent scientific impact as it is actually perceived by experts in their respective field. We run a survey of about 2000 corresponding authors who performed a pairwise impact assessment task across more than 20,000 scientific articles. Results of the survey show that citation data and perceived impact do not align well, unless one properly accounts for psychological biases that affect the opinions of experts with respect to their own papers vs. those of others. Researchers tend to prefer their own publications to the most cited papers in their field of research. There is only a mild positive correlation between the number of citations of top-cited papers and expert preference in pairwise comparisons. This also applies to pairs of papers with several orders of magnitude differences in their total number of accumulated citations. However, when researchers were asked to choose among pairs of their own papers, thus eliminating the bias favouring one's own papers over those of others, they did systematically prefer the most cited article. We conclude that, when scientists have full information and are making unbiased choices, expert opinion on impact is congruent with citation numbers.