IDEAS home Printed from https://ideas.repec.org/a/gam/jpubli/v8y2020i2p34-d373852.html
   My bibliography  Save this article

An Empirical Test of the Inter-Relationships between Various Bibliometric Creative Scholarship Indicators

Author

Listed:
  • Boris Forthmann

    (Institute of Psychology in Education, University of Münster, 48149 Münster, Germany)

  • Mark A. Runco

    (Innovation and Leadership Faculty, Southern Oregon University, Medford, OR 97501, USA)

Abstract

Quantifying the creative quality of scholarly work is a difficult challenge, and, unsurprisingly, empirical research in this area is scarce. This investigation builds on the theoretical distinction between impact (e.g., citation counts) and creative quality (e.g., originality) and extends recent work on using objective measures to assess the originality of scientific publications. Following extensive evidence from creativity research and theoretical deliberations, we operationalized multiple indicators of openness and idea density for bibliometric research. Results showed that in two large bibliometric datasets (creativity research: N = 1643; bibliometrics dataset: N = 2986) correlations between impact and the various indicators for openness, idea density, and originality were negligible to small; this finding supports the discriminant validity of the new creative scholarship indicators. The convergent validity of these indicators was not as clear, but correlations were comparable to previous research on bibliometric originality. Next, we explored the nomological net of various operationalizations of openness and idea density by means of exploratory graph analysis. The openness indicators of variety (based on cited journals and cited first authors) were found to be made up of strongly connected nodes in a separate cluster; the idea density indicators (those based on abstracts or titles of scientific work) also formed a separate cluster. Based on these findings, we discuss the problems arising from the potential methodological overlap among indicators and we offer future directions for bibliometric explorations of the creative quality of scientific publications.

Suggested Citation

  • Boris Forthmann & Mark A. Runco, 2020. "An Empirical Test of the Inter-Relationships between Various Bibliometric Creative Scholarship Indicators," Publications, MDPI, vol. 8(2), pages 1-16, June.
    • Handle: RePEc:gam:jpubli:v:8:y:2020:i:2:p:34-:d:373852
    as

    Download full text from publisher

    File URL: https://www.mdpi.com/2304-6775/8/2/34/pdf
    Download Restriction: no
    File URL: https://www.mdpi.com/2304-6775/8/2/34/
    Download Restriction: no
    ---><---

    References listed on IDEAS

    as
    1. Lingfei Wu & Dashun Wang & James A. Evans, 2019. "Large teams develop and small teams disrupt science and technology," Nature, Nature, vol. 566(7744), pages 378-382, February.
    2. Aria, Massimo & Cuccurullo, Corrado, 2017. "bibliometrix: An R-tool for comprehensive science mapping analysis," Journal of Informetrics, Elsevier, vol. 11(4), pages 959-975.
    3. James Hartley, 2017. "Authors and their citations: a point of view," Scientometrics, Springer;Akadémiai Kiadó, vol. 110(2), pages 1081-1084, February.
    4. Freese, Jeremy & Peterson, David, 2017. "Replication in Social Science," SocArXiv 5bck9, Center for Open Science.
    5. van Buuren, Stef & Groothuis-Oudshoorn, Karin, 2011. "mice: Multivariate Imputation by Chained Equations in R," Journal of Statistical Software, Foundation for Open Access Statistics, vol. 45(i03).
    6. Thomas Heinze & Gerrit Bauer, 2007. "Characterizing creative scientists in nano-S&T: Productivity, multidisciplinarity, and network brokerage in a longitudinal perspective," Scientometrics, Springer;Akadémiai Kiadó, vol. 70(3), pages 811-830, March.
    7. Pierre Azoulay & Joshua S. Graff Zivin & Gustavo Manso, 2011. "Incentives and creativity: evidence from the academic life sciences," RAND Journal of Economics, RAND Corporation, vol. 42(3), pages 527-554, September.
    8. Wang, Jian, 2016. "Knowledge creation in collaboration networks: Effects of tie configuration," Research Policy, Elsevier, vol. 45(1), pages 68-80.
    9. Brown, Ld & Gardner, Jc, 1985. "Using Citation Analysis To Assess The Impact Of Journals And Articles On Contemporary Accounting Research (Car)," Journal of Accounting Research, John Wiley & Sons, Ltd., vol. 23(1), pages 84-109.
    10. Mortaza Jamshidian & Siavash Jalal, 2010. "Tests of Homoscedasticity, Normality, and Missing Completely at Random for Incomplete Multivariate Data," Psychometrika, Springer;The Psychometric Society, vol. 75(4), pages 649-674, December.
    11. Wolfgang Glänzel & Henk F. Moed, 2002. "Journal impact measures in bibliometric research," Scientometrics, Springer;Akadémiai Kiadó, vol. 53(2), pages 171-193, February.
    12. Andy Stirling, 2007. "A General Framework for Analysing Diversity in Science, Technology and Society," SPRU Working Paper Series 156, SPRU - Science Policy Research Unit, University of Sussex Business School.
    13. Adriano S. Melo & Luis Mauricio Bini & Priscilla Carvalho, 2006. "Brazilian articles in international journals on Limnology," Scientometrics, Springer;Akadémiai Kiadó, vol. 67(2), pages 187-199, May.
    14. Russell J. Funk & Jason Owen-Smith, 2017. "A Dynamic Network Measure of Technological Change," Management Science, INFORMS, vol. 63(3), pages 791-817, March.
    15. Lin Zhang & Ronald Rousseau & Wolfgang Glänzel, 2016. "Diversity of references as an indicator of the interdisciplinarity of journals: Taking similarity between subject fields into account," Journal of the Association for Information Science & Technology, Association for Information Science & Technology, vol. 67(5), pages 1257-1265, May.
    16. Hudson F Golino & Sacha Epskamp, 2017. "Exploratory graph analysis: A new approach for estimating the number of dimensions in psychological research," PLOS ONE, Public Library of Science, vol. 12(6), pages 1-26, June.
    17. Sanjay K. Arora & Alan L. Porter & Jan Youtie & Philip Shapira, 2013. "Capturing new developments in an emerging technology: an updated search strategy for identifying nanotechnology research outputs," Scientometrics, Springer;Akadémiai Kiadó, vol. 95(1), pages 351-370, April.
    18. Forthmann, Boris & Jendryczko, David & Scharfen, Jana & Kleinkorres, Ruben & Benedek, Mathias & Holling, Heinz, 2019. "Creative ideation, broad retrieval ability, and processing speed: A confirmatory study of nested cognitive abilities," Intelligence, Elsevier, vol. 75(C), pages 59-72.
    19. Thomas Heinze, 2013. "Creative accomplishments in science: definition, theoretical considerations, examples from science history, and bibliometric findings," Scientometrics, Springer;Akadémiai Kiadó, vol. 95(3), pages 927-940, June.
    20. Sotaro Shibayama & Jian Wang, 2020. "Measuring originality in science," Scientometrics, Springer;Akadémiai Kiadó, vol. 122(1), pages 409-427, January.
    21. András Schubert, 2010. "A reference-based Hirschian similarity measure for journals," Scientometrics, Springer;Akadémiai Kiadó, vol. 84(1), pages 133-147, July.
    22. Alfredo Yegros-Yegros & Ismael Rafols & Pablo D’Este, 2015. "Does Interdisciplinary Research Lead to Higher Citation Impact? The Different Effect of Proximal and Distal Interdisciplinarity," PLOS ONE, Public Library of Science, vol. 10(8), pages 1-21, August.
    23. Franceschet, Massimo & Costantini, Antonio, 2010. "The effect of scholar collaboration on impact and quality of academic papers," Journal of Informetrics, Elsevier, vol. 4(4), pages 540-553.
    24. Golino, Hudson F. & Demetriou, Andreas, 2017. "Estimating the dimensionality of intelligence like data using Exploratory Graph Analysis," Intelligence, Elsevier, vol. 62(C), pages 54-70.
    25. Jesper W. Schneider, 2018. "NHST is still logically flawed," Scientometrics, Springer;Akadémiai Kiadó, vol. 115(1), pages 627-635, April.
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Ziyan Zhang & Junyan Zhang & Pushi Wang, 2024. "Measurement of disruptive innovation and its validity based on improved disruption index," Scientometrics, Springer;Akadémiai Kiadó, vol. 129(11), pages 6477-6531, November.
    2. Christian Leibel & Lutz Bornmann, 2024. "What do we know about the disruption index in scientometrics? An overview of the literature," Scientometrics, Springer;Akadémiai Kiadó, vol. 129(1), pages 601-639, January.

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Shiji Chen & Yanan Guo & Alvin Shijie Ding & Yanhui Song, 2024. "Is interdisciplinarity more likely to produce novel or disruptive research?," Scientometrics, Springer;Akadémiai Kiadó, vol. 129(5), pages 2615-2632, May.
    2. Xin Liu & Yi Bu & Ming Li & Jiang Li, 2024. "Monodisciplinary collaboration disrupts science more than multidisciplinary collaboration," Journal of the Association for Information Science & Technology, Association for Information Science & Technology, vol. 75(1), pages 59-78, January.
    3. Sotaro Shibayama & Jian Wang, 2020. "Measuring originality in science," Scientometrics, Springer;Akadémiai Kiadó, vol. 122(1), pages 409-427, January.
    4. Hou, Jianhua & Wang, Dongyi & Li, Jing, 2022. "A new method for measuring the originality of academic articles based on knowledge units in semantic networks," Journal of Informetrics, Elsevier, vol. 16(3).
    5. Deng Cheng & Zhang Xue & Yang Zhibo & Zhang Mingze, 2025. "Impact of interdisciplinarity on disruptive innovation: the moderating role of collaboration pattern and collaboration size," Scientometrics, Springer;Akadémiai Kiadó, vol. 130(4), pages 2379-2401, April.
    6. Tang, Kun & Li, Baiyang & Zhu, Qiyu & Ma, Lecun, 2024. "Disruptive content, cross agglomeration interaction, and agglomeration replacement: Does cohesion foster strength?," Journal of Informetrics, Elsevier, vol. 18(4).
    7. Shiji Chen & Yanhui Song & Fei Shu & Vincent Larivière, 2022. "Interdisciplinarity and impact: the effects of the citation time window," Scientometrics, Springer;Akadémiai Kiadó, vol. 127(5), pages 2621-2642, May.
    8. Jiang, Huihuang & Zhou, Jianlin & Ding, Yiming & Zeng, An, 2024. "Overcoming recognition delays in disruptive research: The impact of team size, familiarity, and reputation," Journal of Informetrics, Elsevier, vol. 18(4).
    9. Yang, Alex J., 2024. "On the temporal diversity of knowledge in science," Journal of Informetrics, Elsevier, vol. 18(4).
    10. Seolmin Yang & So Young Kim, 2023. "Knowledge-integrated research is more disruptive when supported by homogeneous funding sources: a case of US federally funded research in biomedical and life sciences," Scientometrics, Springer;Akadémiai Kiadó, vol. 128(6), pages 3257-3282, June.
    11. Yang, Alex J., 2024. "Unveiling the impact and dual innovation of funded research," Journal of Informetrics, Elsevier, vol. 18(1).
    12. Lu Liu & Benjamin F. Jones & Brian Uzzi & Dashun Wang, 2023. "Data, measurement and empirical methods in the science of science," Nature Human Behaviour, Nature, vol. 7(7), pages 1046-1058, July.
    13. Hou, Jianhua & Li, Hao & Zhang, Yang, 2024. "Influence of interdisciplinarity of scientific papers on the durability of citation diffusion: A perspective from citation discontinuance," Journal of Informetrics, Elsevier, vol. 18(3).
    14. Shiji Chen & Kaiqi Zhang & Junping Qiu & Jiaqi Chai, 2024. "Interdisciplinarity and expert rating: an analysis based on faculty opinions," Scientometrics, Springer;Akadémiai Kiadó, vol. 129(11), pages 6597-6628, November.
    15. Lina Xu & Steven Dellaportas & Zhiqiang Yang & Jin Wang, 2023. "More on the relationship between interdisciplinary accounting research and citation impact," Accounting and Finance, Accounting and Finance Association of Australia and New Zealand, vol. 63(4), pages 4779-4803, December.
    16. Sam Arts & Nicola Melluso & Reinhilde Veugelers, 2023. "Beyond Citations: Measuring Novel Scientific Ideas and their Impact in Publication Text," Papers 2309.16437, arXiv.org, revised Dec 2024.
    17. Zhentao Liang & Jin Mao & Gang Li, 2023. "Bias against scientific novelty: A prepublication perspective," Journal of the Association for Information Science & Technology, Association for Information Science & Technology, vol. 74(1), pages 99-114, January.
    18. Zhang, Jia & Wang, Jian & Winnink, Jos & Jong, Simcha, 2025. "Collaboration networks and radical innovation: Two faces of tie strength and structural holes," Journal of Informetrics, Elsevier, vol. 19(1).
    19. Li, Meiling & Wang, Yang & Du, Haifeng & Bai, Aruhan, 2024. "Motivating innovation: The impact of prestigious talent funding on junior scientists," Research Policy, Elsevier, vol. 53(9).
    20. Wolfgang Glänzel & Koenraad Debackere, 2022. "Various aspects of interdisciplinarity in research and how to quantify and measure those," Scientometrics, Springer;Akadémiai Kiadó, vol. 127(9), pages 5551-5569, September.

    More about this item

    Keywords

    ;
    ;
    ;
    ;
    ;

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:gam:jpubli:v:8:y:2020:i:2:p:34-:d:373852. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: MDPI Indexing Manager (email available below). General contact details of provider: https://www.mdpi.com .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.