IDEAS home Printed from https://ideas.repec.org/a/eee/infome/v16y2022i2s1751157722000426.html
   My bibliography  Save this article

Metrics and mechanisms: Measuring the unmeasurable in the science of science

Author

Listed:
  • Wu, Lingfei
  • Kittur, Aniket
  • Youn, Hyejin
  • Milojević, Staša
  • Leahey, Erin
  • Fiore, Stephen M.
  • Ahn, Yong-Yeol

Abstract

What science does, what science could do, and how to make science work? If we want to know the answers to these questions, we need to be able to uncover the mechanisms of science, going beyond metrics that are easily collectible and quantifiable. In this perspective piece, we link metrics to mechanisms by demonstrating how emerging metrics of science not only offer complementaries to existing ones, but also shed light on the hidden structure and mechanisms of science. Based on fundamental properties of science, we classify existing theories and findings into: hot and cold science referring to attention shift between scientific fields, fast and slow science reflecting productivity of scientists and teams, soft and hard science revealing reproducibility of scientific research. We suggest that interest about mechanisms of science since Derek J. de Solla Price, Robert K. Merton, Eugene Garfield, and many others complement the zeitgeist in pursuing new, complex metrics without understanding the underlying processes. We propose that understanding and modeling the mechanisms of science condition effective development and application of metrics.

Suggested Citation

  • Wu, Lingfei & Kittur, Aniket & Youn, Hyejin & Milojević, Staša & Leahey, Erin & Fiore, Stephen M. & Ahn, Yong-Yeol, 2022. "Metrics and mechanisms: Measuring the unmeasurable in the science of science," Journal of Informetrics, Elsevier, vol. 16(2).
    • Handle: RePEc:eee:infome:v:16:y:2022:i:2:s1751157722000426
    DOI: 10.1016/j.joi.2022.101290
    as

    Download full text from publisher

    File URL: http://www.sciencedirect.com/science/article/pii/S1751157722000426
    Download Restriction: Full text for ScienceDirect subscribers only
    File URL: https://libkey.io/10.1016/j.joi.2022.101290?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    As the access to this document is restricted, you may want to search for a different version of it.

    References listed on IDEAS

    as
    1. Katherine W. McCain, 2011. "Eponymy and Obliteration by Incorporation: The case of the “Nash Equilibrium”," Journal of the American Society for Information Science and Technology, Association for Information Science & Technology, vol. 62(7), pages 1412-1424, July.
    2. Nicholas Bloom & Charles I. Jones & John Van Reenen & Michael Webb, 2020. "Are Ideas Getting Harder to Find?," American Economic Review, American Economic Association, vol. 110(4), pages 1104-1144, April.
    3. Yian Yin & Yuxiao Dong & Kuansan Wang & Dashun Wang & Benjamin Jones, 2021. "Science as a Public Good: Public Use and Funding of Science," NBER Working Papers 28748, National Bureau of Economic Research, Inc.
    4. Milojević, Staša, 2015. "Quantifying the cognitive extent of science," Journal of Informetrics, Elsevier, vol. 9(4), pages 962-973.
    5. Nelson, Richard R., 2003. "On the uneven evolution of human know-how," Research Policy, Elsevier, vol. 32(6), pages 909-922, June.
    6. Lin, Yiling & Evans, James A. & Wu, Lingfei, 2022. "New directions in science emerge from disconnection and discord," Journal of Informetrics, Elsevier, vol. 16(1).
    7. Wang, Jian & Veugelers, Reinhilde & Stephan, Paula, 2017. "Bias against novelty in science: A cautionary tale for users of bibliometric indicators," Research Policy, Elsevier, vol. 46(8), pages 1416-1436.
    8. Christian Catalini & Christian Fons-Rosen & Patrick Gaulé, 2020. "How Do Travel Costs Shape Collaboration?," Management Science, INFORMS, vol. 66(8), pages 3340-3360, August.
    9. Russell J. Funk & Jason Owen-Smith, 2017. "A Dynamic Network Measure of Technological Change," Management Science, INFORMS, vol. 63(3), pages 791-817, March.
    10. Haeussler, Carolin & Sauermann, Henry, 2020. "Division of labor in collaborative knowledge production: The role of team size and interdisciplinarity," Research Policy, Elsevier, vol. 49(6).
    11. Walsh, John P. & Lee, You-Na, 2015. "The bureaucratization of science," Research Policy, Elsevier, vol. 44(8), pages 1584-1600.
    12. Chen, Chaomei & Song, Min & Heo, Go Eun, 2018. "A scalable and adaptive method for finding semantically equivalent cue words of uncertainty," Journal of Informetrics, Elsevier, vol. 12(1), pages 158-180.
    13. Yin, Yian & Wang, Dashun, 2017. "The time dimension of science: Connecting the past to the future," Journal of Informetrics, Elsevier, vol. 11(2), pages 608-621.
    14. Shouhuai Xu & Moti Yung & Jingguo Wang, 2021. "Seeking Foundations for the Science of Cyber Security," Information Systems Frontiers, Springer, vol. 23(2), pages 263-267, April.
    15. Lu Liu & Yang Wang & Roberta Sinatra & C. Lee Giles & Chaoming Song & Dashun Wang, 2018. "Hot streaks in artistic, cultural, and scientific careers," Nature, Nature, vol. 559(7714), pages 396-399, July.
    16. Bas Hofstra & Vivek V. Kulkarni & Sebastian Munoz-Najar Galvez & Bryan He & Dan Jurafsky & Daniel A. McFarland, 2020. "The Diversity–Innovation Paradox in Science," Proceedings of the National Academy of Sciences, Proceedings of the National Academy of Sciences, vol. 117(17), pages 9284-9291, April.
    17. Staša Milojević & Filippo Radicchi & John P. Walsh, 2018. "Changing demographics of scientific careers: The rise of the temporary workforce," Proceedings of the National Academy of Sciences, Proceedings of the National Academy of Sciences, vol. 115(50), pages 12616-12623, December.
    18. Chen, Jiyao & Shao, Diana & Fan, Shaokun, 2021. "Destabilization and consolidation: Conceptualizing, measuring, and validating the dual characteristics of technology," Research Policy, Elsevier, vol. 50(1).
    19. Pierre Azoulay & Christian Fons-Rosen & Joshua S. Graff Zivin, 2019. "Does Science Advance One Funeral at a Time?," American Economic Review, American Economic Association, vol. 109(8), pages 2889-2920, August.
    20. Michele Coscia & Frank M. H. Neffke & Ricardo Hausmann, 2020. "Knowledge diffusion in the network of international business travel," Nature Human Behaviour, Nature, vol. 4(10), pages 1011-1020, October.
    21. Lingfei Wu & Dashun Wang & James A. Evans, 2019. "Large teams develop and small teams disrupt science and technology," Nature, Nature, vol. 566(7744), pages 378-382, February.
    22. James G. March, 1991. "Exploration and Exploitation in Organizational Learning," Organization Science, INFORMS, vol. 2(1), pages 71-87, February.
    23. Cassidy R. Sugimoto, 2021. "Scientific success by numbers," Nature, Nature, vol. 593(7857), pages 30-31, May.
    24. Kevin W. Boyack & Richard Klavans & Katy Börner, 2005. "Mapping the backbone of science," Scientometrics, Springer;Akadémiai Kiadó, vol. 64(3), pages 351-374, August.
    25. C. A. Hidalgo & B. Klinger & A. -L. Barabasi & R. Hausmann, 2007. "The Product Space Conditions the Development of Nations," Papers 0708.2090, arXiv.org.
    26. Kevin J. Boudreau & Nicola Lacetera & Karim R. Lakhani, 2011. "Incentives and Problem Uncertainty in Innovation Contests: An Empirical Analysis," Management Science, INFORMS, vol. 57(5), pages 843-863, May.
    27. Staša Milojević, 2012. "How Are Academic Age, Productivity and Collaboration Related to Citing Behavior of Researchers?," PLOS ONE, Public Library of Science, vol. 7(11), pages 1-13, November.
    28. Vahe Tshitoyan & John Dagdelen & Leigh Weston & Alexander Dunn & Ziqin Rong & Olga Kononova & Kristin A. Persson & Gerbrand Ceder & Anubhav Jain, 2019. "Unsupervised word embeddings capture latent knowledge from materials science literature," Nature, Nature, vol. 571(7763), pages 95-98, July.
    29. Antonoyiannakis, Manolis, 2018. "Impact Factors and the Central Limit Theorem: Why citation averages are scale dependent," Journal of Informetrics, Elsevier, vol. 12(4), pages 1072-1088.
    30. Marcus R. Munafò & Brian A. Nosek & Dorothy V. M. Bishop & Katherine S. Button & Christopher D. Chambers & Nathalie Percie du Sert & Uri Simonsohn & Eric-Jan Wagenmakers & Jennifer J. Ware & John P. A, 2017. "A manifesto for reproducible science," Nature Human Behaviour, Nature, vol. 1(1), pages 1-9, January.
    31. Shalabh, 2021. "Statistical inference via data science," Journal of the Royal Statistical Society Series A, Royal Statistical Society, vol. 184(3), pages 1155-1155, July.
    32. Diana Hicks & Paul Wouters & Ludo Waltman & Sarah de Rijcke & Ismael Rafols, 2015. "Bibliometrics: The Leiden Manifesto for research metrics," Nature, Nature, vol. 520(7548), pages 429-431, April.
    33. Katherine W. McCain, 2011. "Eponymy and Obliteration by Incorporation: The case of the “Nash Equilibrium”," Journal of the Association for Information Science & Technology, Association for Information Science & Technology, vol. 62(7), pages 1412-1424, July.
    34. Abhay S. D. Rajput & Sangeeta Sharma, 2021. "India: draft science policy calls for public engagement," Nature, Nature, vol. 592(7852), pages 26-26, April.
    35. Jean J. Wang & Sarah X. Shao & Fred Y. Ye, 2021. "Identifying 'seed' papers in sciences," Scientometrics, Springer;Akadémiai Kiadó, vol. 126(7), pages 6001-6011, July.
    36. Mikko Packalen & Jay Bhattacharya, 2015. "Age and the Trying Out of New Ideas," NBER Working Papers 20920, National Bureau of Economic Research, Inc.
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Hayat D. Bedru & Chen Zhang & Feng Xie & Shuo Yu & Iftikhar Hussain, 2023. "CLARA: citation and similarity-based author ranking," Scientometrics, Springer;Akadémiai Kiadó, vol. 128(2), pages 1091-1117, February.
    2. Pierre Pelletier & Kevin Wirtz, 2023. "Sails and Anchors: The Complementarity of Exploratory and Exploitative Scientists in Knowledge Creation," Papers 2312.10476, arXiv.org.

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Lu Liu & Benjamin F. Jones & Brian Uzzi & Dashun Wang, 2023. "Data, measurement and empirical methods in the science of science," Nature Human Behaviour, Nature, vol. 7(7), pages 1046-1058, July.
    2. Lin, Yiling & Evans, James A. & Wu, Lingfei, 2022. "New directions in science emerge from disconnection and discord," Journal of Informetrics, Elsevier, vol. 16(1).
    3. Pierre Pelletier & Kevin Wirtz, 2023. "Sails and Anchors: The Complementarity of Exploratory and Exploitative Scientists in Knowledge Creation," Papers 2312.10476, arXiv.org.
    4. Liang, Zhentao & Ba, Zhichao & Mao, Jin & Li, Gang, 2023. "Research complexity increases with scientists’ academic age: Evidence from library and information science," Journal of Informetrics, Elsevier, vol. 17(1).
    5. Malte Hückstädt, 2023. "Ten reasons why research collaborations succeed—a random forest approach," Scientometrics, Springer;Akadémiai Kiadó, vol. 128(3), pages 1923-1950, March.
    6. Cinzia Daraio & Simone Di Leo & Loet Leydesdorff, 2022. "Using the Leiden Rankings as a Heuristics: Evidence from Italian universities in the European landscape," LEM Papers Series 2022/08, Laboratory of Economics and Management (LEM), Sant'Anna School of Advanced Studies, Pisa, Italy.
    7. Yue Wang & Ning Li & Bin Zhang & Qian Huang & Jian Wu & Yang Wang, 2023. "The effect of structural holes on producing novel and disruptive research in physics," Scientometrics, Springer;Akadémiai Kiadó, vol. 128(3), pages 1801-1823, March.
    8. Thomas, Duncan Andrew & Ramos-Vielba, Irene, 2022. "Reframing study of research(er) funding towards configurations and trails," SocArXiv uty2v, Center for Open Science.
    9. Yiling Lin & Carl Benedikt Frey & Lingfei Wu, 2022. "Remote Collaboration Fuses Fewer Breakthrough Ideas," Papers 2206.01878, arXiv.org, revised Oct 2023.
    10. Cinzia Daraio & Simone Di Leo & Loet Leydesdorff, 2023. "A heuristic approach based on Leiden rankings to identify outliers: evidence from Italian universities in the European landscape," Scientometrics, Springer;Akadémiai Kiadó, vol. 128(1), pages 483-510, January.
    11. Katchanov, Yurij L. & Markova, Yulia V. & Shmatko, Natalia A., 2023. "Uncited papers in the structure of scientific communication," Journal of Informetrics, Elsevier, vol. 17(2).
    12. Jiang, Zhuoren & Lin, Tianqianjin & Huang, Cui, 2023. "Deep representation learning of scientific paper reveals its potential scholarly impact," Journal of Informetrics, Elsevier, vol. 17(1).
    13. Libo Sheng & Dongqing Lyu & Xuanmin Ruan & Hongquan Shen & Ying Cheng, 2023. "The association between prior knowledge and the disruption of an article," Scientometrics, Springer;Akadémiai Kiadó, vol. 128(8), pages 4731-4751, August.
    14. Marek Kwiek & Wojciech Roszka, 2022. "Academic vs. biological age in research on academic careers: a large-scale study with implications for scientifically developing systems," Scientometrics, Springer;Akadémiai Kiadó, vol. 127(6), pages 3543-3575, June.
    15. Zhuanlan Sun & C. Clark Cao & Sheng Liu & Yiwei Li & Chao Ma, 2024. "Behavioral consequences of second-person pronouns in written communications between authors and reviewers of scientific papers," Nature Communications, Nature, vol. 15(1), pages 1-12, December.
    16. Liu, Meijun & Jaiswal, Ajay & Bu, Yi & Min, Chao & Yang, Sijie & Liu, Zhibo & Acuña, Daniel & Ding, Ying, 2022. "Team formation and team impact: The balance between team freshness and repeat collaboration," Journal of Informetrics, Elsevier, vol. 16(4).
    17. Haeussler, Carolin & Sauermann, Henry, 2020. "Division of labor in collaborative knowledge production: The role of team size and interdisciplinarity," Research Policy, Elsevier, vol. 49(6).
    18. Hou, Jianhua & Wang, Dongyi & Li, Jing, 2022. "A new method for measuring the originality of academic articles based on knowledge units in semantic networks," Journal of Informetrics, Elsevier, vol. 16(3).
    19. Weihua Li & Sam Zhang & Zhiming Zheng & Skyler J. Cranmer & Aaron Clauset, 2022. "Untangling the network effects of productivity and prominence among scientists," Nature Communications, Nature, vol. 13(1), pages 1-11, December.
    20. Liyin Zhang & Yuchen Qian & Chao Ma & Jiang Li, 2023. "Continued collaboration shortens the transition period of scientists who move to another institution," Scientometrics, Springer;Akadémiai Kiadó, vol. 128(3), pages 1765-1784, March.

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:eee:infome:v:16:y:2022:i:2:s1751157722000426. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Catherine Liu (email available below). General contact details of provider: http://www.elsevier.com/locate/joi .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.