IDEAS home Printed from https://ideas.repec.org/a/spr/scient/v126y2021i9d10.1007_s11192-021-04048-0.html
   My bibliography  Save this article

Are bibliometric measures consistent with scientists’ perceptions? The case of interdisciplinarity in research

Author

Listed:
  • Alfonso Ávila-Robinson

    (Tecnologico de Monterrey)

  • Cristian Mejia

    (Tokyo Institute of Technology)

  • Shintaro Sengoku

    (Tokyo Institute of Technology)

Abstract

Interdisciplinary research (IDR) has become an important component in the conduction of leading-edge science and innovation. From the different approaches available to measuring IDR, bibliometric indicators have experienced the greatest growth. Despite the frequent use of bibliometric measures of IDR in research and policymaking, their adequacy has not been validated against scientists’ perceptions. Using the case of an IDR-oriented research institute in Japan, this study aims to investigate the differences and similarities between the outcomes of common bibliometric measures of IDR and the scientists’ perceptions of IDR. We used a unique dataset combining bibliometric measures with survey data collected from the scientists’ self-assessment of their research. This study also investigates the factors influencing the outcomes of bibliometrics and scientists' perceptions. Moreover, this study explores how IDR qualitative and quantitative measures differ from those that are more intuitive, such as scientific impact. It was observed that there is no “holy grail” measure for interdisciplinarity when compared with scientific impact, for which the impact factor is considered as a key metric by scientists. While bibliometric measures of interdisciplinarity show mild correlations with scientists' perceptions, they display high discriminatory power. The disagreement between qualitative and quantitative evaluations, as well as the significant field-specific nature of interdisciplinarity, calls for the use of multidimensional assessment approaches for assessing IDR, and the building of a consensus about the meaning and measurement of interdisciplinarity among scientists, respectively. The results of this study provide a series of guidelines for a more effective implementation of interdisciplinarity-oriented R&D policies at different organizational levels.

Suggested Citation

  • Alfonso Ávila-Robinson & Cristian Mejia & Shintaro Sengoku, 2021. "Are bibliometric measures consistent with scientists’ perceptions? The case of interdisciplinarity in research," Scientometrics, Springer;Akadémiai Kiadó, vol. 126(9), pages 7477-7502, September.
    • Handle: RePEc:spr:scient:v:126:y:2021:i:9:d:10.1007_s11192-021-04048-0
    DOI: 10.1007/s11192-021-04048-0
    as

    Download full text from publisher

    File URL: http://link.springer.com/10.1007/s11192-021-04048-0
    File Function: Abstract
    Download Restriction: Access to the full text of the articles in this series is restricted.
    File URL: https://libkey.io/10.1007/s11192-021-04048-0?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    As the access to this document is restricted, you may want to search for a different version of it.

    References listed on IDEAS

    as
    1. Vincent Larivière & Yves Gingras, 2010. "On the relationship between interdisciplinarity and scientific impact," Journal of the Association for Information Science & Technology, Association for Information Science & Technology, vol. 61(1), pages 126-131, January.
    2. Alexis-Michel Mugabushaka & Anthi Kyriakou & Theo Papazoglou, 2016. "Bibliometric indicators of interdisciplinarity: the potential of the Leinster–Cobbold diversity indices to study disciplinary diversity," Scientometrics, Springer;Akadémiai Kiadó, vol. 107(2), pages 593-607, May.
    3. Loet Leydesdorff & Caroline S. Wagner & Lutz Bornmann, 2018. "Betweenness and diversity in journal citation networks as measures of interdisciplinarity—A tribute to Eugene Garfield," Scientometrics, Springer;Akadémiai Kiadó, vol. 114(2), pages 567-592, February.
    4. Lin Zhang & Beibei Sun & Zaida Chinchilla-Rodríguez & Lixin Chen & Ying Huang, 2018. "Interdisciplinarity and collaboration: on the relationship between disciplinary diversity in departmental affiliations and reference lists," Scientometrics, Springer;Akadémiai Kiadó, vol. 117(1), pages 271-291, October.
    5. Vincent Larivière & Yves Gingras, 2010. "On the relationship between interdisciplinarity and scientific impact," Journal of the American Society for Information Science and Technology, Association for Information Science & Technology, vol. 61(1), pages 126-131, January.
    6. Leydesdorff, Loet & Rafols, Ismael, 2011. "Indicators of the interdisciplinarity of journals: Diversity, centrality, and citations," Journal of Informetrics, Elsevier, vol. 5(1), pages 87-100.
    7. Andy Stirling, 2007. "A General Framework for Analysing Diversity in Science, Technology and Society," SPRU Working Paper Series 156, SPRU - Science Policy Research Unit, University of Sussex Business School.
    8. Giovanni Abramo & Ciriaco Andrea D’Angelo & Flavia Costa, 2017. "Do interdisciplinary research teams deliver higher gains to science?," Scientometrics, Springer;Akadémiai Kiadó, vol. 111(1), pages 317-336, April.
    9. Ludo Waltman & Nees Jan Eck, 2012. "A new methodology for constructing a publication-level classification system of science," Journal of the Association for Information Science & Technology, Association for Information Science & Technology, vol. 63(12), pages 2378-2392, December.
    10. Fernanda Morillo & María Bordons & Isabel Gómez, 2001. "An approach to interdisciplinarity through bibliometric indicators," Scientometrics, Springer;Akadémiai Kiadó, vol. 51(1), pages 203-222, April.
    11. Richard Klavans & Kevin W. Boyack, 2017. "Which Type of Citation Analysis Generates the Most Accurate Taxonomy of Scientific and Technical Knowledge?," Journal of the Association for Information Science & Technology, Association for Information Science & Technology, vol. 68(4), pages 984-998, April.
    12. Lin Zhang & Ronald Rousseau & Wolfgang Glänzel, 2016. "Diversity of references as an indicator of the interdisciplinarity of journals: Taking similarity between subject fields into account," Journal of the Association for Information Science & Technology, Association for Information Science & Technology, vol. 67(5), pages 1257-1265, May.
    13. Pablo Jensen & Katsiaryna Lutkouskaya, 2014. "The many dimensions of laboratories’ interdisciplinarity," Scientometrics, Springer;Akadémiai Kiadó, vol. 98(1), pages 619-631, January.
    14. M. M. Kessler, 1963. "Bibliographic coupling between scientific papers," American Documentation, Wiley Blackwell, vol. 14(1), pages 10-25, January.
    15. Jian Wang & Bart Thijs & Wolfgang Glänzel, 2015. "Interdisciplinarity and Impact: Distinct Effects of Variety, Balance, and Disparity," PLOS ONE, Public Library of Science, vol. 10(5), pages 1-18, May.
    16. Ismael Rafols & Martin Meyer, 2010. "Diversity and network coherence as indicators of interdisciplinarity: case studies in bionanoscience," Scientometrics, Springer;Akadémiai Kiadó, vol. 82(2), pages 263-287, February.
    17. Abramo, Giovanni & D’Angelo, Ciriaco Andrea & Zhang, Lin, 2018. "A comparison of two approaches for measuring interdisciplinary research output: The disciplinary diversity of authors vs the disciplinary diversity of the reference list," Journal of Informetrics, Elsevier, vol. 12(4), pages 1182-1193.
    18. Arsia Amir Aslani & Vincent Mangematin, 2010. "The future of drug discovery and development: Shifting emphasis towards personalized medicine," Grenoble Ecole de Management (Post-Print) hal-00749148, HAL.
    19. Wagner, Caroline S. & Roessner, J. David & Bobb, Kamau & Klein, Julie Thompson & Boyack, Kevin W. & Keyton, Joann & Rafols, Ismael & Börner, Katy, 2011. "Approaches to understanding and measuring interdisciplinary scientific research (IDR): A review of the literature," Journal of Informetrics, Elsevier, vol. 5(1), pages 14-26.
    20. Martín-Martín, Alberto & Orduna-Malea, Enrique & Thelwall, Mike & Delgado López-Cózar, Emilio, 2018. "Google Scholar, Web of Science, and Scopus: A systematic comparison of citations in 252 subject categories," Journal of Informetrics, Elsevier, vol. 12(4), pages 1160-1177.
    21. Huutoniemi, Katri & Klein, Julie Thompson & Bruun, Henrik & Hukkinen, Janne, 2010. "Analyzing interdisciplinarity: Typology and indicators," Research Policy, Elsevier, vol. 39(1), pages 79-88, February.
    22. Ludo Waltman & Nees Jan van Eck, 2012. "A new methodology for constructing a publication‐level classification system of science," Journal of the American Society for Information Science and Technology, Association for Information Science & Technology, vol. 63(12), pages 2378-2392, December.
    23. Alan L. Porter & Alex S. Cohen & J. David Roessner & Marty Perreault, 2007. "Measuring researcher interdisciplinarity," Scientometrics, Springer;Akadémiai Kiadó, vol. 72(1), pages 117-147, July.
    24. Ávila-Robinson, Alfonso & Miyazaki, Kumiko, 2013. "Dynamics of scientific knowledge bases as proxies for discerning technological emergence — The case of MEMS/NEMS technologies," Technological Forecasting and Social Change, Elsevier, vol. 80(6), pages 1071-1084.
    25. Giovanni Abramo & Ciriaco Andrea D'Angelo & Flavia Di Costa, 2012. "Identifying interdisciplinarity through the disciplinary classification of coauthors of scientific publications," Journal of the American Society for Information Science and Technology, Association for Information Science & Technology, vol. 63(11), pages 2206-2222, November.
    26. Giovanni Abramo & Ciriaco Andrea D'Angelo & Flavia Costa, 2012. "Identifying interdisciplinarity through the disciplinary classification of coauthors of scientific publications," Journal of the Association for Information Science & Technology, Association for Information Science & Technology, vol. 63(11), pages 2206-2222, November.
    27. Silva, F.N. & Rodrigues, F.A. & Oliveira, O.N. & da F. Costa, L., 2013. "Quantifying the interdisciplinarity of scientific journals and fields," Journal of Informetrics, Elsevier, vol. 7(2), pages 469-477.
    28. Ruiz-Castillo, Javier & Waltman, Ludo, 2015. "Field-normalized citation impact indicators using algorithmically constructed classification systems of science," Journal of Informetrics, Elsevier, vol. 9(1), pages 102-117.
    29. Luis Sanz-Menéndez & María Bordons & M Angeles Zulueta, 2001. "Interdisciplinarity as a multidimensional concept: its measure in three different research areas," Research Evaluation, Oxford University Press, vol. 10(1), pages 47-58, April.
    30. Arsia Amir Aslani & Vincent Mangematin, 2010. "The future of drug discovery and development: Shifting emphasis towards personalized medicine," Post-Print hal-00749148, HAL.
    31. Joachim Schummer, 2004. "Multidisciplinarity, interdisciplinarity, and patterns of research collaboration in nanoscience and nanotechnology," Scientometrics, Springer;Akadémiai Kiadó, vol. 59(3), pages 425-465, March.
    32. Alfredo Yegros-Yegros & Ismael Rafols & Pablo D’Este, 2015. "Does Interdisciplinary Research Lead to Higher Citation Impact? The Different Effect of Proximal and Distal Interdisciplinarity," PLOS ONE, Public Library of Science, vol. 10(8), pages 1-21, August.
    33. Rafols, Ismael & Leydesdorff, Loet & O’Hare, Alice & Nightingale, Paul & Stirling, Andy, 2012. "How journal rankings can suppress interdisciplinary research: A comparison between Innovation Studies and Business & Management," Research Policy, Elsevier, vol. 41(7), pages 1262-1282.
    34. Leydesdorff, Loet & Wagner, Caroline S. & Bornmann, Lutz, 2019. "Interdisciplinarity as diversity in citation patterns among journals: Rao-Stirling diversity, relative variety, and the Gini coefficient," Journal of Informetrics, Elsevier, vol. 13(1), pages 255-269.
    35. Alan L. Porter & Ismael Rafols, 2009. "Is science becoming more interdisciplinary? Measuring and mapping six research fields over time," Scientometrics, Springer;Akadémiai Kiadó, vol. 81(3), pages 719-745, December.
    36. Loet Leydesdorff, 2007. "Betweenness centrality as an indicator of the interdisciplinarity of scientific journals," Journal of the American Society for Information Science and Technology, Association for Information Science & Technology, vol. 58(9), pages 1303-1319, July.
    37. Alan L Porter & David J Roessner & Anne E Heberger, 2008. "How interdisciplinary is a given body of research?," Research Evaluation, Oxford University Press, vol. 17(4), pages 273-282, December.
    38. Matthew Claudel & Emanuele Massaro & Paolo Santi & Fiona Murray & Carlo Ratti, 2017. "An exploration of collaborative scientific production at MIT through spatial organization and institutional affiliation," PLOS ONE, Public Library of Science, vol. 12(6), pages 1-22, June.
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Jingjing Ren & Fang Wang & Minglu Li, 2023. "Dynamics and characteristics of interdisciplinary research in scientific breakthroughs: case studies of Nobel-winning research in the past 120 years," Scientometrics, Springer;Akadémiai Kiadó, vol. 128(8), pages 4383-4419, August.

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Abramo, Giovanni & D’Angelo, Ciriaco Andrea & Zhang, Lin, 2018. "A comparison of two approaches for measuring interdisciplinary research output: The disciplinary diversity of authors vs the disciplinary diversity of the reference list," Journal of Informetrics, Elsevier, vol. 12(4), pages 1182-1193.
    2. Lin Zhang & Beibei Sun & Zaida Chinchilla-Rodríguez & Lixin Chen & Ying Huang, 2018. "Interdisciplinarity and collaboration: on the relationship between disciplinary diversity in departmental affiliations and reference lists," Scientometrics, Springer;Akadémiai Kiadó, vol. 117(1), pages 271-291, October.
    3. Wolfgang Glänzel & Koenraad Debackere, 2022. "Various aspects of interdisciplinarity in research and how to quantify and measure those," Scientometrics, Springer;Akadémiai Kiadó, vol. 127(9), pages 5551-5569, September.
    4. Shiji Chen & Yanhui Song & Fei Shu & Vincent Larivière, 2022. "Interdisciplinarity and impact: the effects of the citation time window," Scientometrics, Springer;Akadémiai Kiadó, vol. 127(5), pages 2621-2642, May.
    5. Hongyu Zhou & Raf Guns & Tim C. E. Engels, 2022. "Are social sciences becoming more interdisciplinary? Evidence from publications 1960–2014," Journal of the Association for Information Science & Technology, Association for Information Science & Technology, vol. 73(9), pages 1201-1221, September.
    6. Chen, Shiji & Qiu, Junping & Arsenault, Clément & Larivière, Vincent, 2021. "Exploring the interdisciplinarity patterns of highly cited papers," Journal of Informetrics, Elsevier, vol. 15(1).
    7. Jingjing Ren & Fang Wang & Minglu Li, 2023. "Dynamics and characteristics of interdisciplinary research in scientific breakthroughs: case studies of Nobel-winning research in the past 120 years," Scientometrics, Springer;Akadémiai Kiadó, vol. 128(8), pages 4383-4419, August.
    8. Xuefeng Wang & Zhinan Wang & Ying Huang & Yun Chen & Yi Zhang & Huichao Ren & Rongrong Li & Jinhui Pang, 2017. "Measuring interdisciplinarity of a research system: detecting distinction between publication categories and citation categories," Scientometrics, Springer;Akadémiai Kiadó, vol. 111(3), pages 2023-2039, June.
    9. Wooseok Jang & Heeyeul Kwon & Yongtae Park & Hakyeon Lee, 2018. "Predicting the degree of interdisciplinarity in academic fields: the case of nanotechnology," Scientometrics, Springer;Akadémiai Kiadó, vol. 116(1), pages 231-254, July.
    10. Sander Zwanenburg & Maryam Nakhoda & Peter Whigham, 2022. "Toward greater consistency and validity in measuring interdisciplinarity: a systematic and conceptual evaluation," Scientometrics, Springer;Akadémiai Kiadó, vol. 127(12), pages 7769-7788, December.
    11. Shengli Deng & Sudi Xia, 2020. "Mapping the interdisciplinarity in information behavior research: a quantitative study using diversity measure and co-occurrence analysis," Scientometrics, Springer;Akadémiai Kiadó, vol. 124(1), pages 489-513, July.
    12. Giovanni Abramo & Ciriaco Andrea D’Angelo & Flavia Costa, 2017. "Specialization versus diversification in research activities: the extent, intensity and relatedness of field diversification by individual scientists," Scientometrics, Springer;Akadémiai Kiadó, vol. 112(3), pages 1403-1418, September.
    13. Lina Xu & Steven Dellaportas & Jin Wang, 2022. "A study of interdisciplinary accounting research: analysing the diversity of cited references," Accounting and Finance, Accounting and Finance Association of Australia and New Zealand, vol. 62(2), pages 2131-2162, June.
    14. Ricardo Arencibia-Jorge & Rosa Lidia Vega-Almeida & José Luis Jiménez-Andrade & Humberto Carrillo-Calvet, 2022. "Evolutionary stages and multidisciplinary nature of artificial intelligence research," Scientometrics, Springer;Akadémiai Kiadó, vol. 127(9), pages 5139-5158, September.
    15. Ugo Moschini & Elena Fenialdi & Cinzia Daraio & Giancarlo Ruocco & Elisa Molinari, 2020. "A comparison of three multidisciplinarity indices based on the diversity of Scopus subject areas of authors’ documents, their bibliography and their citing papers," Scientometrics, Springer;Akadémiai Kiadó, vol. 125(2), pages 1145-1158, November.
    16. Rafols, Ismael & Leydesdorff, Loet & O’Hare, Alice & Nightingale, Paul & Stirling, Andy, 2012. "How journal rankings can suppress interdisciplinary research: A comparison between Innovation Studies and Business & Management," Research Policy, Elsevier, vol. 41(7), pages 1262-1282.
    17. Qing Ke, 2023. "Interdisciplinary research and technological impact: evidence from biomedicine," Scientometrics, Springer;Akadémiai Kiadó, vol. 128(4), pages 2035-2077, April.
    18. Leydesdorff, Loet & Wagner, Caroline S. & Bornmann, Lutz, 2019. "Interdisciplinarity as diversity in citation patterns among journals: Rao-Stirling diversity, relative variety, and the Gini coefficient," Journal of Informetrics, Elsevier, vol. 13(1), pages 255-269.
    19. Loet Leydesdorff & Caroline S. Wagner & Lutz Bornmann, 2018. "Betweenness and diversity in journal citation networks as measures of interdisciplinarity—A tribute to Eugene Garfield," Scientometrics, Springer;Akadémiai Kiadó, vol. 114(2), pages 567-592, February.
    20. Juste Raimbault, 2019. "Exploration of an interdisciplinary scientific landscape," Scientometrics, Springer;Akadémiai Kiadó, vol. 119(2), pages 617-641, May.

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:spr:scient:v:126:y:2021:i:9:d:10.1007_s11192-021-04048-0. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Sonal Shukla or Springer Nature Abstracting and Indexing (email available below). General contact details of provider: http://www.springer.com .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.