IDEAS home Printed from https://ideas.repec.org/a/eee/infome/v19y2025i4s1751157725001026.html

Model uncertainty in the evaluation of the impact of interdisciplinary research in Business Studies: A Multiverse Analysis

Author

Listed:
  • Cantone, Giulio Giacomo
  • Nightingale, Paul

Abstract

There are many operational definitions and many indicators of interdisciplinary research. As a consequence, claims about its scientific impact may suffer from high model uncertainty and low credibility, since they can be based on specific results highly dependent on implicit choices of the modelling process. This study addresses the issue with Multiverse Analysis, a protocol for multi-model inference that specifies the modelling factors that can generate variability in the results. Combinations of these factors are fit and analysed simultaneously, so that inference cannot be selective in reporting results. 1,344 regression models are fit to a sample of 5,828 articles from journals of Business Studies. The Multiverse Analysis does not find support for the claim that interdisciplinary articles of Business are more cited. Even though a minority of the models reach statistical significance, the median estimates for the effect size are close to zero. Through an analysis of variance it is established that the choice of the indicator is the most influential, reproducing 45% of the total variance. This result confirms that claims on the scientific impact of IDR are highly dependent on the operational definition. The choice of the sources of metadata on articles, a factor previously overlooked in literature, reproduce 10% of the total variance. Findings suggest caution in accepting generic claims about the scientific impact of interdisciplinary research in social sciences.

Suggested Citation

  • Cantone, Giulio Giacomo & Nightingale, Paul, 2025. "Model uncertainty in the evaluation of the impact of interdisciplinary research in Business Studies: A Multiverse Analysis," Journal of Informetrics, Elsevier, vol. 19(4).
    • Handle: RePEc:eee:infome:v:19:y:2025:i:4:s1751157725001026
    DOI: 10.1016/j.joi.2025.101740
    as

    Download full text from publisher

    File URL: http://www.sciencedirect.com/science/article/pii/S1751157725001026
    Download Restriction: Full text for ScienceDirect subscribers only
    File URL: https://libkey.io/10.1016/j.joi.2025.101740?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    As the access to this document is restricted, you may want to

    for a different version of it.

    References listed on IDEAS

    as
    1. Sander Zwanenburg & Maryam Nakhoda & Peter Whigham, 2022. "Toward greater consistency and validity in measuring interdisciplinarity: a systematic and conceptual evaluation," Scientometrics, Springer;Akadémiai Kiadó, vol. 127(12), pages 7769-7788, December.
    2. Chen, Shiji & Qiu, Junping & Arsenault, Clément & Larivière, Vincent, 2021. "Exploring the interdisciplinarity patterns of highly cited papers," Journal of Informetrics, Elsevier, vol. 15(1).
    3. Giulio Giacomo Cantone & Venera Tomaselli, 2024. "Theory and methods of the multiverse: an application for panel-based models," Quality & Quantity: International Journal of Methodology, Springer, vol. 58(2), pages 1447-1480, April.
    4. Manuel Goyanes & Márton Demeter & Aurea Grané & Irene Albarrán-Lozano & Homero Gil de Zúñiga, 2020. "A mathematical approach to assess research diversity: operationalization and applicability in communication sciences, political science, and beyond," Scientometrics, Springer;Akadémiai Kiadó, vol. 125(3), pages 2299-2322, December.
    5. Buckley, Peter J. & Chapman, Malcolm, 1996. "Theory and method in international business research," International Business Review, Elsevier, vol. 5(3), pages 233-245, June.
    6. Lili Wang & Ad Notten & Alexandru Surpatean, 2013. "Interdisciplinarity of nano research fields: a keyword mining approach," Scientometrics, Springer;Akadémiai Kiadó, vol. 94(3), pages 877-892, March.
    7. Shiji Chen & Yanhui Song & Fei Shu & Vincent Larivière, 2022. "Interdisciplinarity and impact: the effects of the citation time window," Scientometrics, Springer;Akadémiai Kiadó, vol. 127(5), pages 2621-2642, May.
    8. Nightingale, Paul, 1998. "A cognitive model of innovation," Research Policy, Elsevier, vol. 27(7), pages 689-709, November.
    9. Jian Wang & Bart Thijs & Wolfgang Glänzel, 2015. "Interdisciplinarity and Impact: Distinct Effects of Variety, Balance, and Disparity," PLOS ONE, Public Library of Science, vol. 10(5), pages 1-18, May.
    10. Leahey, Erin & Barringer, Sondra N., 2020. "Universities’ commitment to interdisciplinary research: To what end?," Research Policy, Elsevier, vol. 49(2).
    11. Robert L. Whited, 2025. "Institutional Ties and Publications in Top Accounting Journals," Management Science, INFORMS, vol. 71(11), pages 9041-9070, November.
    12. Nicholas Bloom & Charles I. Jones & John Van Reenen & Michael Webb, 2020. "Are Ideas Getting Harder to Find?," American Economic Review, American Economic Association, vol. 110(4), pages 1104-1144, April.
    13. Ismael Rafols & Martin Meyer, 2010. "Diversity and network coherence as indicators of interdisciplinarity: case studies in bionanoscience," Scientometrics, Springer;Akadémiai Kiadó, vol. 82(2), pages 263-287, February.
    14. Yi Bu & Mengyang Li & Weiye Gu & Win‐bin Huang, 2021. "Topic diversity: A discipline scheme‐free diversity measurement for journals," Journal of the Association for Information Science & Technology, Association for Information Science & Technology, vol. 72(5), pages 523-539, May.
    15. Lin Zhang & Beibei Sun & Lidan Jiang & Ying Huang, 2021. "On the relationship between interdisciplinarity and impact: Distinct effects on academic and broader impact [A Comparison of Two Approaches for Measuring Interdisciplinary Research Output: The Disciplinary Diversity of Authors vs the Disciplinary ," Research Evaluation, Oxford University Press, vol. 30(3), pages 256-268.
    16. Paolo Girardi & Anna Vesely & Daniël Lakens & Gianmarco Altoè & Massimiliano Pastore & Antonio Calcagnì & Livio Finos, 2024. "Post-selection Inference in Multiverse Analysis (PIMA): An Inferential Framework Based on the Sign Flipping Score Test," Psychometrika, Springer;The Psychometric Society, vol. 89(2), pages 542-568, June.
    17. Keisuke Okamura, 2019. "Interdisciplinarity revisited: evidence for research impact and dynamism," Humanities and Social Sciences Communications, Palgrave Macmillan, vol. 5(1), pages 1-9, December.
    18. Bornmann, Lutz & Tekles, Alexander & Zhang, Helena H. & Ye, Fred Y., 2019. "Do we measure novelty when we analyze unusual combinations of cited references? A validation study of bibliometric novelty indicators based on F1000Prime data," Journal of Informetrics, Elsevier, vol. 13(4).
    19. Ronald Rousseau, 2018. "The repeat rate: from Hirschman to Stirling," Scientometrics, Springer;Akadémiai Kiadó, vol. 116(1), pages 645-653, July.
    20. Bethany K Laursen & Nicole Motzer & Kelly J Anderson, 2022. "Pathways for assessing interdisciplinarity: A systematic review," Research Evaluation, Oxford University Press, vol. 31(3), pages 326-343.
    21. Kim, Hyeyoung & Park, Hyelin & Song, Min, 2022. "Developing a topic-driven method for interdisciplinarity analysis," Journal of Informetrics, Elsevier, vol. 16(2).
    22. repec:osf:metaar:hva4p_v1 is not listed on IDEAS
    23. Wolfgang Glänzel & Koenraad Debackere, 2022. "Various aspects of interdisciplinarity in research and how to quantify and measure those," Scientometrics, Springer;Akadémiai Kiadó, vol. 127(9), pages 5551-5569, September.
    24. Alfredo Yegros-Yegros & Ismael Rafols & Pablo D’Este, 2015. "Does Interdisciplinary Research Lead to Higher Citation Impact? The Different Effect of Proximal and Distal Interdisciplinarity," PLOS ONE, Public Library of Science, vol. 10(8), pages 1-21, August.
    25. Giulio Giacomo Cantone & Er-Te Zheng & Venera Tomaselli & Paul Nightingale, 2025. "Estimation of disciplinary similarity with large language models," Scientometrics, Springer;Akadémiai Kiadó, vol. 130(10), pages 5345-5373, October.
    26. Alessio Lachi & Josep Llach & Jordi Perramon & Michela Baccini & Andrea Saltelli, 2025. "Robustification of structural equation modelling via global sensitivity analysis," Statistical Methods & Applications, Springer;Società Italiana di Statistica, vol. 34(2), pages 211-236, May.
    27. Hou, Jianhua & Li, Hao & Zhang, Yang, 2024. "Influence of interdisciplinarity of scientific papers on the durability of citation diffusion: A perspective from citation discontinuance," Journal of Informetrics, Elsevier, vol. 18(3).
    28. Andy Stirling, 2007. "A General Framework for Analysing Diversity in Science, Technology and Society," SPRU Working Paper Series 156, SPRU - Science Policy Research Unit, University of Sussex Business School.
    29. Giulio Giacomo Cantone & Venera Tomaselli, 2024. "A Multiversal Model of Vibration of Effects of the Equitable and Sustainable Well-Being (BES) on Fertility," Social Indicators Research: An International and Interdisciplinary Journal for Quality-of-Life Measurement, Springer, vol. 175(3), pages 941-964, December.
    30. Hongyu Zhou & Raf Guns & Tim C. E. Engels, 2022. "Are social sciences becoming more interdisciplinary? Evidence from publications 1960–2014," Journal of the Association for Information Science & Technology, Association for Information Science & Technology, vol. 73(9), pages 1201-1221, September.
    31. Wang, Jian & Veugelers, Reinhilde & Stephan, Paula, 2017. "Bias against novelty in science: A cautionary tale for users of bibliometric indicators," Research Policy, Elsevier, vol. 46(8), pages 1416-1436.
    32. Fontana, Magda & Iori, Martina & Montobbio, Fabio & Sinatra, Roberta, 2020. "New and atypical combinations: An assessment of novelty and interdisciplinarity," Research Policy, Elsevier, vol. 49(7).
    33. Abramo, Giovanni & D’Angelo, Ciriaco Andrea & Zhang, Lin, 2018. "A comparison of two approaches for measuring interdisciplinary research output: The disciplinary diversity of authors vs the disciplinary diversity of the reference list," Journal of Informetrics, Elsevier, vol. 12(4), pages 1182-1193.
    34. Lin Zhang & Beibei Sun & Fei Shu & Ying Huang, 2022. "Comparing paper level classifications across different methods and systems: an investigation of Nature publications," Scientometrics, Springer;Akadémiai Kiadó, vol. 127(12), pages 7633-7651, December.
    35. Silva, Filipi N. & Amancio, Diego R. & Bardosova, Maria & Costa, Luciano da F. & Oliveira, Osvaldo N., 2016. "Using network science and text analytics to produce surveys in a scientific topic," Journal of Informetrics, Elsevier, vol. 10(2), pages 487-502.
    36. Sondra N. Barringer & Erin Leahey & Karina Salazar, 2020. "What Catalyzes Research Universities to Commit to Interdisciplinary Research?," Research in Higher Education, Springer;Association for Institutional Research, vol. 61(6), pages 679-705, September.
    37. Abramo, Giovanni & D’Angelo, Ciriaco Andrea, 2015. "The relationship between the number of authors of a publication, its citations and the impact factor of the publishing journal: Evidence from Italy," Journal of Informetrics, Elsevier, vol. 9(4), pages 746-761.
    38. Rüdiger Mutz, 2022. "Diversity and interdisciplinarity: Should variety, balance and disparity be combined as a product or better as a sum? An information-theoretical and statistical estimation approach," Scientometrics, Springer;Akadémiai Kiadó, vol. 127(12), pages 7397-7414, December.
    39. Vogel, Rick & Hattke, Fabian & Petersen, Jessica, 2017. "Journal rankings in management and business studies: What rules do we play by?," Research Policy, Elsevier, vol. 46(10), pages 1707-1722.
    40. Aisling J. Daly & Jan M. Baetens & Bernard De Baets, 2018. "Ecological Diversity: Measuring the Unmeasurable," Mathematics, MDPI, vol. 6(7), pages 1-28, July.
    41. Leydesdorff, Loet & Wagner, Caroline S. & Bornmann, Lutz, 2019. "Interdisciplinarity as diversity in citation patterns among journals: Rao-Stirling diversity, relative variety, and the Gini coefficient," Journal of Informetrics, Elsevier, vol. 13(1), pages 255-269.
    42. Alan L. Porter & Ismael Rafols, 2009. "Is science becoming more interdisciplinary? Measuring and mapping six research fields over time," Scientometrics, Springer;Akadémiai Kiadó, vol. 81(3), pages 719-745, December.
    43. Giulio Giacomo Cantone, 2024. "How to measure interdisciplinary research? A systemic design for the model of measurement," Scientometrics, Springer;Akadémiai Kiadó, vol. 129(8), pages 4937-4982, August.
    44. Megan L Head & Luke Holman & Rob Lanfear & Andrew T Kahn & Michael D Jennions, 2015. "The Extent and Consequences of P-Hacking in Science," PLOS Biology, Public Library of Science, vol. 13(3), pages 1-15, March.
    45. Cantone, Giulio Giacomo, 2024. "How to measure interdisciplinary research? A systematic, yet critical, review," MetaArXiv hva4p, Center for Open Science.
    46. Wagner, Caroline S. & Roessner, J. David & Bobb, Kamau & Klein, Julie Thompson & Boyack, Kevin W. & Keyton, Joann & Rafols, Ismael & Börner, Katy, 2011. "Approaches to understanding and measuring interdisciplinary scientific research (IDR): A review of the literature," Journal of Informetrics, Elsevier, vol. 5(1), pages 14-26.
    47. Rafols, Ismael & Leydesdorff, Loet & O’Hare, Alice & Nightingale, Paul & Stirling, Andy, 2012. "How journal rankings can suppress interdisciplinary research: A comparison between Innovation Studies and Business & Management," Research Policy, Elsevier, vol. 41(7), pages 1262-1282.
    48. Rui Dai & Lawrence Donohue & Qingyi (Freda) Drechsler & Wei Jiang, 2023. "Dissemination, Publication, and Impact of Finance Research: When Novelty Meets Conventionality," Review of Finance, European Finance Association, vol. 27(1), pages 79-141.
    49. Christian Leibel & Lutz Bornmann, 2024. "Specification uncertainty: what the disruption index tells us about the (hidden) multiverse of bibliometric indicators," Scientometrics, Springer;Akadémiai Kiadó, vol. 129(12), pages 7971-7979, December.
    50. Bryan Kelly & Dimitris Papanikolaou & Amit Seru & Matt Taddy, 2021. "Measuring Technological Innovation over the Long Run," American Economic Review: Insights, American Economic Association, vol. 3(3), pages 303-320, September.
    Full references (including those not matched with items on IDEAS)

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Giulio Giacomo Cantone, 2024. "How to measure interdisciplinary research? A systemic design for the model of measurement," Scientometrics, Springer;Akadémiai Kiadó, vol. 129(8), pages 4937-4982, August.
    2. Jingjing Ren & Fang Wang & Minglu Li, 2023. "Dynamics and characteristics of interdisciplinary research in scientific breakthroughs: case studies of Nobel-winning research in the past 120 years," Scientometrics, Springer;Akadémiai Kiadó, vol. 128(8), pages 4383-4419, August.
    3. Shiji Chen & Yanan Guo & Alvin Shijie Ding & Yanhui Song, 2024. "Is interdisciplinarity more likely to produce novel or disruptive research?," Scientometrics, Springer;Akadémiai Kiadó, vol. 129(5), pages 2615-2632, May.
    4. Jin, Qianqian & Chen, Hongshu & Wang, Xuefeng & Liu, Jia, 2026. "Do synthesis programs facilitate interdisciplinary research? Evidence from the NSFC’s Major Research Plan," Research Policy, Elsevier, vol. 55(2).
    5. Maryam Nakhoda & Peter Whigham & Sander Zwanenburg, 2023. "Quantifying and addressing uncertainty in the measurement of interdisciplinarity," Scientometrics, Springer;Akadémiai Kiadó, vol. 128(11), pages 6107-6127, November.
    6. Shiji Chen & Yanhui Song & Fei Shu & Vincent Larivière, 2022. "Interdisciplinarity and impact: the effects of the citation time window," Scientometrics, Springer;Akadémiai Kiadó, vol. 127(5), pages 2621-2642, May.
    7. Xiaojing Cai & Xiaozan Lyu & Ping Zhou, 2023. "The relationship between interdisciplinarity and citation impact—a novel perspective on citation accumulation," Humanities and Social Sciences Communications, Palgrave Macmillan, vol. 10(1), pages 1-12, December.
    8. Andrea Bonaccorsi & Nicola Melluso & Francesco Alessandro Massucci, 2022. "Exploring the antecedents of interdisciplinarity at the European Research Council: a topic modeling approach," Scientometrics, Springer;Akadémiai Kiadó, vol. 127(12), pages 6961-6991, December.
    9. Zhang, Yujie & He, Guoxiu & Jiang, Zhuoren, 2025. "Mapping science and revealing disciplinary communication modalities via pre-trained graph neural networks," Journal of Informetrics, Elsevier, vol. 19(4).
    10. Junping Qiu & Yunlong Yu & Shiji Chen & Teng Zhao & Shanshan Wang, 2024. "Effect of Scientific Collaboration on Interdisciplinarity in Climate Change From a Scientometric Perspective," SAGE Open, , vol. 14(2), pages 21582440241, April.
    11. Xin Liu & Yi Bu & Ming Li & Jiang Li, 2024. "Monodisciplinary collaboration disrupts science more than multidisciplinary collaboration," Journal of the Association for Information Science & Technology, Association for Information Science & Technology, vol. 75(1), pages 59-78, January.
    12. Alfonso Ávila-Robinson & Cristian Mejia & Shintaro Sengoku, 2021. "Are bibliometric measures consistent with scientists’ perceptions? The case of interdisciplinarity in research," Scientometrics, Springer;Akadémiai Kiadó, vol. 126(9), pages 7477-7502, September.
    13. Chen, Shiji & Qiu, Junping & Arsenault, Clément & Larivière, Vincent, 2021. "Exploring the interdisciplinarity patterns of highly cited papers," Journal of Informetrics, Elsevier, vol. 15(1).
    14. Deng Cheng & Zhang Xue & Yang Zhibo & Zhang Mingze, 2025. "Impact of interdisciplinarity on disruptive innovation: the moderating role of collaboration pattern and collaboration size," Scientometrics, Springer;Akadémiai Kiadó, vol. 130(4), pages 2379-2401, April.
    15. Seolmin Yang & So Young Kim, 2023. "Knowledge-integrated research is more disruptive when supported by homogeneous funding sources: a case of US federally funded research in biomedical and life sciences," Scientometrics, Springer;Akadémiai Kiadó, vol. 128(6), pages 3257-3282, June.
    16. Du, Anbang & Head, Michael & Brede, Markus, 2025. "Integration vs segregation: Network analysis of interdisciplinarity in funded and unfunded research on infectious diseases," Journal of Informetrics, Elsevier, vol. 19(1).
    17. Shiji Chen & Kaiqi Zhang & Junping Qiu & Jiaqi Chai, 2024. "Interdisciplinarity and expert rating: an analysis based on faculty opinions," Scientometrics, Springer;Akadémiai Kiadó, vol. 129(11), pages 6597-6628, November.
    18. Fei Shu & Jesse David Dinneen & Shiji Chen, 2022. "Measuring the disparity among scientific disciplines using Library of Congress Subject Headings," Scientometrics, Springer;Akadémiai Kiadó, vol. 127(6), pages 3613-3628, June.
    19. Wu, Keye & Sun, Jianjun & Wang, Jiajie & Kang, Lele, 2025. "How does science convergence influence technology convergence? Different impacts of science-push and technology-pull," Technological Forecasting and Social Change, Elsevier, vol. 215(C).
    20. Cheng, Deng & Xue, Zhang, 2025. "Exploring the relationship between interdisciplinarity and scientific breakthrough speed: a study based on Nobel Prize-winning papers," Journal of Informetrics, Elsevier, vol. 19(3).

    More about this item

    Keywords

    ;
    ;
    ;
    ;
    ;

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:eee:infome:v:19:y:2025:i:4:s1751157725001026. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Catherine Liu (email available below). General contact details of provider: http://www.elsevier.com/locate/joi .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.