Reliance on science by inventors: Hybrid extraction of in‐text patent‐to‐article citations

We curate and characterize a complete set of citations from patents to scientific articles, including 16.8 million from the full text of USPTO and EPO patents. Combining hand‐tuned heuristics and the GROBID machine‐learning package, we achieve much higher performance than machine learning alone. Recall is evaluated with a set of 5939 randomly sampled, cross‐verified “known good” citations, which the authors have never seen. At 99.4% precision, we achieve recall rates of 78% for the full test set and 88% for references specified without mistakes. We compare these “in‐text” citations with those on the front page of patents. In‐text citations are more diverse temporally, geographically, and topically; moreover, they are less self‐referential and less likely to be copied from one patent to the next. In‐text citations have dropped from two‐thirds of all patent‐to‐article citations half a century ago to about one‐third today. In replicating two articles that use only front‐page citations, we show that failing to capture in‐text citations leads to understating the role of academic science in commercial invention. All patent‐to‐article citations, the known‐good test set, and the source code are available at http://relianceonscience.org.