IDEAS home Printed from https://ideas.repec.org/a/spr/scient/v101y2014i1d10.1007_s11192-014-1337-0.html
   My bibliography  Save this article

Can the technological impact of academic journals be evaluated? The practice of non-patent reference (NPR) analysis

Author

Listed:
  • Yi-Ching Liaw

    (Ming Chi University of Technology)

  • Te-Yi Chan

    (Science & Technology Policy Research and Information Center)

  • Chin-Yuan Fan

    (Science & Technology Policy Research and Information Center)

  • Cheng-Hsin Chiang

    (Macau University of Science and Technology)

Abstract

Journal rankings and journal ratings are important to governments, research institutes, and scientific research in general, and they frequently serve as the criteria for evaluating research performance to determine whether specific researchers will receive promotions and/or earn research grants. However, the only widely adopted journal assessment method is known as impact factor (IF), which focuses on citations in academic journals. However, IF disregards the technological applications and value of academic journals. In this article, we propose a method to rank academic journals that utilizes non-patent references in patent documents. We also compare the differences between journal rankings derived by using IF with those derived from the Intellectual Property Citation Index (IPCI) across different fields; moreover, some fields contain positive and significant correlations between IF and the IPCI. The results of this study offer a new perspective from which to assess the technological value of academic journals, particularly those in the technological and scientific fields. This study considers linkages among science and technology and the needs of the stakeholders in journal assessment to shed light on journal assessment and journal ranking methods.

Suggested Citation

  • Yi-Ching Liaw & Te-Yi Chan & Chin-Yuan Fan & Cheng-Hsin Chiang, 2014. "Can the technological impact of academic journals be evaluated? The practice of non-patent reference (NPR) analysis," Scientometrics, Springer;Akadémiai Kiadó, vol. 101(1), pages 17-37, October.
    • Handle: RePEc:spr:scient:v:101:y:2014:i:1:d:10.1007_s11192-014-1337-0
    DOI: 10.1007/s11192-014-1337-0
    as

    Download full text from publisher

    File URL: http://link.springer.com/10.1007/s11192-014-1337-0
    File Function: Abstract
    Download Restriction: Access to the full text of the articles in this series is restricted.
    File URL: https://libkey.io/10.1007/s11192-014-1337-0?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    As the access to this document is restricted, you may want to search for a different version of it.

    References listed on IDEAS

    as
    1. Kothari, Tanvi & Lahiri, Somnath, 2012. "Yesterday, today and tomorrow: An overview of research publications in the Journal of International Management," Journal of International Management, Elsevier, vol. 18(1), pages 102-110.
    2. Chiang Kao & Hsiou-Wei Lin & San-Lin Chung & Wei-Chi Tsai & Jyh-Shen Chiou & Yen-Liang Chen & Liang-Hsuan Chen & Shih-Chieh Fang & Hwei-Lan Pao, 2008. "Ranking Taiwanese management journals: A case study," Scientometrics, Springer;Akadémiai Kiadó, vol. 76(1), pages 95-115, July.
    3. Michael D. Gordon, 1982. "Citation ranking versus subjective evaluation in the determination of journal hierachies in the social sciences," Journal of the American Society for Information Science, Association for Information Science & Technology, vol. 33(1), pages 55-57, January.
    4. Jones, M. J. & Brinn, T. & Pendlebury, M., 1996. "Journal evaluation methodologies: A balanced response," Omega, Elsevier, vol. 24(5), pages 607-612, October.
    5. Benjamin M. Althouse & Jevin D. West & Carl T. Bergstrom & Theodore Bergstrom, 2009. "Differences in impact factor across fields and over time," Journal of the American Society for Information Science and Technology, Association for Information Science & Technology, vol. 60(1), pages 27-34, January.
    6. McMillan, G. Steven & Narin, Francis & Deeds, David L., 2000. "An analysis of the critical role of public science in innovation: the case of biotechnology," Research Policy, Elsevier, vol. 29(1), pages 1-8, January.
    7. Arnold Verbeek & Koenraad Debackere & Marc Luwel & Petra Andries & Edwin Zimmermann & Filip Deleus, 2002. "Linking science to technology: Using bibliographic references in patents to build linkage schemes," Scientometrics, Springer;Akadémiai Kiadó, vol. 54(3), pages 399-420, July.
    8. J. A. García & Rosa Rodriguez-Sánchez & J. Fdez-Valdivia, 2011. "Overall prestige of journals with ranking score above a given threshold," Scientometrics, Springer;Akadémiai Kiadó, vol. 89(1), pages 229-243, October.
    9. Ignacio Palacios-Huerta & Oscar Volij, 2004. "The Measurement of Intellectual Influence," Econometrica, Econometric Society, vol. 72(3), pages 963-977, May.
    10. Narin, Francis & Olivastro, Dominic, 1992. "Status report: Linkage between technology and science," Research Policy, Elsevier, vol. 21(3), pages 237-249, June.
    11. Jacques Michel & Bernd Bettels, 2001. "Patent citation analysis.A closer look at the basic input data from patent search reports," Scientometrics, Springer;Akadémiai Kiadó, vol. 51(1), pages 185-201, April.
    12. Philip M. Podsakoff & Scott B. MacKenzie & Daniel G. Bachrach & Nathan P. Podsakoff, 2005. "The influence of management journals in the 1980s and 1990s," Strategic Management Journal, Wiley Blackwell, vol. 26(5), pages 473-488, May.
    13. Jonathan C. Catling & Victoria L. Mason & Dominic Upton, 2009. "Quality is in the eye of the beholder? An evaluation of impact factors and perception of journal prestige in the UK," Scientometrics, Springer;Akadémiai Kiadó, vol. 81(2), pages 333-345, November.
    14. Henk F. Moed & Lisa Colledge & Jan Reedijk & Felix Moya-Anegon & Vicente Guerrero-Bote & Andrew Plume & Mayur Amin, 2012. "Citation-based metrics are appropriate tools in journal assessment provided that they are accurate and used in an informed way," Scientometrics, Springer;Akadémiai Kiadó, vol. 92(2), pages 367-376, August.
    15. Stefano Brusoni & Paola Criscuolo & Aldo Geuna, 2005. "The knowledge bases of the world's largest pharmaceutical groups: what do patent citations to non-patent literature reveal?," Economics of Innovation and New Technology, Taylor & Francis Journals, vol. 14(5), pages 395-415.
    16. Loet Leydesdorff & Lutz Bornmann, 2011. "Integrated impact indicators compared with impact factors: An alternative research design with policy implications," Journal of the Association for Information Science & Technology, Association for Information Science & Technology, vol. 62(11), pages 2133-2146, November.
    17. Sutter, Matthias & Kocher, Martin G., 2001. "Tools for evaluating research output: Are citation-base rankings of economics journals stable?," Munich Reprints in Economics 18222, University of Munich, Department of Economics.
    18. Robert J. W. Tijssen & Martijn S. Visser & Thed N. van Leeuwen, 2002. "Benchmarking international scientific excellence: Are highly cited research papers an appropriate frame of reference?," Scientometrics, Springer;Akadémiai Kiadó, vol. 54(3), pages 381-397, July.
    19. Julie Callaert & Bart Van Looy & Arnold Verbeek & Koenraad Debackere & Bart Thijs, 2006. "Traces of Prior Art: An analysis of non-patent references found in patent documents," Scientometrics, Springer;Akadémiai Kiadó, vol. 69(1), pages 3-20, October.
    20. Giovanni Abramo & Ciriaco Andrea D'Angelo & Flavia Di Costa, 2009. "Mapping Excellence in National Research Systems," Evaluation Review, , vol. 33(2), pages 159-188, April.
    21. Gaby Haddow & Paul Genoni, 2010. "Citation analysis and peer ranking of Australian social science journals," Scientometrics, Springer;Akadémiai Kiadó, vol. 85(2), pages 471-487, November.
    22. Narin, Francis & Hamilton, Kimberly S. & Olivastro, Dominic, 1997. "The increasing linkage between U.S. technology and public science," Research Policy, Elsevier, vol. 26(3), pages 317-330, October.
    23. Julie Callaert & Joris Grouwels & Bart Looy, 2012. "Delineating the scientific footprint in technology: Identifying scientific publications within non-patent references," Scientometrics, Springer;Akadémiai Kiadó, vol. 91(2), pages 383-398, May.
    24. Vastag, Gyula & Montabon, Frank, 2002. "Journal characteristics, rankings and social acculturation in operations management," Omega, Elsevier, vol. 30(2), pages 109-126, April.
    25. Boyd Cohen, 2007. "Journal ratings and footprints: a North American perspective of organizations and the natural environment journal quality," Business Strategy and the Environment, Wiley Blackwell, vol. 16(1), pages 64-74, January.
    26. Narongrit Sombatsompop & Teerasak Markpin, 2005. "Making an equality of ISI impact factors for different subject fields," Journal of the American Society for Information Science and Technology, Association for Information Science & Technology, vol. 56(7), pages 676-683, May.
    27. Huang, Mu-Hsuan & Huang, Wei-Tzu & Chen, Dar-Zen, 2014. "Technological impact factor: An indicator to measure the impact of academic publications on practical innovation," Journal of Informetrics, Elsevier, vol. 8(1), pages 241-251.
    28. Juan Miguel Campanario, 2011. "Large increases and decreases in journal impact factors in only one year: The effect of journal self-citations," Journal of the Association for Information Science & Technology, Association for Information Science & Technology, vol. 62(2), pages 230-235, February.
    29. David Pontille & Didier Torny, 2010. "The controversial policies of journal ratings: evaluating social sciences and humanities," Research Evaluation, Oxford University Press, vol. 19(5), pages 347-360, December.
    30. Xia Gao & Jiancheng Guan, 2009. "Networks of scientific journals: An exploration of Chinese patent data," Scientometrics, Springer;Akadémiai Kiadó, vol. 80(1), pages 283-302, July.
    31. Werner Marx & Lutz Bornmann & Manuel Cardona, 2010. "Reference standards and reference multipliers for the comparison of the citation impact of papers published in different time periods," Journal of the Association for Information Science & Technology, Association for Information Science & Technology, vol. 61(10), pages 2061-2069, October.
    32. Schoenmakers, Wilfred & Duysters, Geert, 2010. "The technological origins of radical inventions," Research Policy, Elsevier, vol. 39(8), pages 1051-1059, October.
    33. Anita Coleman, 2007. "Assessing the value of a journal beyond the impact factor," Journal of the American Society for Information Science and Technology, Association for Information Science & Technology, vol. 58(8), pages 1148-1161, June.
    34. Péter Vinkler, 2002. "Subfield problems in applying the Garfield (Impact) Factors in practice," Scientometrics, Springer;Akadémiai Kiadó, vol. 53(2), pages 267-279, February.
    35. R. J. W. Tussen & R. K. Buter & Th. N. van Leeuwen, 2000. "Technological Relevance of Science: An Assessment of Citation Linkages between Patents and Research Papers," Scientometrics, Springer;Akadémiai Kiadó, vol. 47(2), pages 389-412, February.
    36. Matthias Sutter & Martin G. Kocher, 2001. "Tools for Evaluating Research Output," Evaluation Review, , vol. 25(5), pages 555-566, October.
    37. Martin Meyer, 2002. "Tracing knowledge flows in innovation systems," Scientometrics, Springer;Akadémiai Kiadó, vol. 54(2), pages 193-212, June.
    38. Juan Miguel Campanario, 2011. "Large increases and decreases in journal impact factors in only one year: The effect of journal self‐citations," Journal of the American Society for Information Science and Technology, Association for Information Science & Technology, vol. 62(2), pages 230-235, February.
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Antonio Fernandez-Cano & Inés M. Fernández-Guerrero, 2017. "A multivariate model for evaluating emergency medicine journals," Scientometrics, Springer;Akadémiai Kiadó, vol. 110(2), pages 991-1003, February.
    2. Gazni, Ali, 2020. "The growing number of patent citations to scientific papers: Changes in the world, nations, and fields," Technology in Society, Elsevier, vol. 62(C).
    3. Dejian Yu & Zhaoping Yan, 2022. "Combining machine learning and main path analysis to identify research front: from the perspective of science-technology linkage," Scientometrics, Springer;Akadémiai Kiadó, vol. 127(7), pages 4251-4274, July.
    4. Rongying Zhao & Xinlai Li & Zhisen Liang & Danyang Li, 2019. "Development strategy and collaboration preference in S&T of enterprises based on funded papers: a case study of Google," Scientometrics, Springer;Akadémiai Kiadó, vol. 121(1), pages 323-347, October.
    5. Gómez-Barroso, José Luis & Feijóo, Claudio & Quiles-Casas, Manuel & Bohlin, Erik, 2017. "The evolution of the telecommunications policy agenda: Forty years of articles in Telecommunications Policy," Telecommunications Policy, Elsevier, vol. 41(10), pages 853-877.

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Bar-Ilan, Judit, 2008. "Informetrics at the beginning of the 21st century—A review," Journal of Informetrics, Elsevier, vol. 2(1), pages 1-52.
    2. Gazni, Ali, 2020. "The growing number of patent citations to scientific papers: Changes in the world, nations, and fields," Technology in Society, Elsevier, vol. 62(C).
    3. Leten, Bart & Kelchtermans, Stijn & Belderbos, Ren, 2010. "Internal Basic Research, External Basic Research and the Technological Performance of Pharmaceutical Firms," Working Papers 2010/12, Hogeschool-Universiteit Brussel, Faculteit Economie en Management.
    4. Bart Looy & Tom Magerman & Koenraad Debackere, 2007. "Developing technology in the vicinity of science: An examination of the relationship between science intensity (of patents) and technological productivity within the field of biotechnology," Scientometrics, Springer;Akadémiai Kiadó, vol. 70(2), pages 441-458, February.
    5. Acosta, Manuel & Coronado, Daniel, 2003. "Science-technology flows in Spanish regions: An analysis of scientific citations in patents," Research Policy, Elsevier, vol. 32(10), pages 1783-1803, December.
    6. Pajić, Dejan, 2015. "On the stability of citation-based journal rankings," Journal of Informetrics, Elsevier, vol. 9(4), pages 990-1006.
    7. Leonardo Costa Ribeiro & Glenda Kruss & Gustavo Britto & Américo Tristão Bernardes & Eduardo Motta e Albuquerque, 2014. "A methodology for unveiling global innovation networks: patent citations as clues to cross border knowledge flows," Scientometrics, Springer;Akadémiai Kiadó, vol. 101(1), pages 61-83, October.
    8. Julie Callaert & Maikel Pellens & Bart Looy, 2014. "Sources of inspiration? Making sense of scientific references in patents," Scientometrics, Springer;Akadémiai Kiadó, vol. 98(3), pages 1617-1629, March.
    9. Martin Meyer & Kevin Grant & Piera Morlacchi & Dagmara Weckowska, 2014. "Triple Helix indicators as an emergent area of enquiry: a bibliometric perspective," Scientometrics, Springer;Akadémiai Kiadó, vol. 99(1), pages 151-174, April.
    10. Amalia Mas-Bleda & Mike Thelwall, 2016. "Can alternative indicators overcome language biases in citation counts? A comparison of Spanish and UK research," Scientometrics, Springer;Akadémiai Kiadó, vol. 109(3), pages 2007-2030, December.
    11. Chen, Lixin, 2017. "Do patent citations indicate knowledge linkage? The evidence from text similarities between patents and their citations," Journal of Informetrics, Elsevier, vol. 11(1), pages 63-79.
    12. Wang, Jean J. & Ye, Fred Y., 2021. "Probing into the interactions between papers and patents of new CRISPR/CAS9 technology: A citation comparison," Journal of Informetrics, Elsevier, vol. 15(4).
    13. Yashuang Qi & Na Zhu & Yujia Zhai & Ying Ding, 2018. "The mutually beneficial relationship of patents and scientific literature: topic evolution in nanoscience," Scientometrics, Springer;Akadémiai Kiadó, vol. 115(2), pages 893-911, May.
    14. Jerome K. Vanclay, 2012. "Impact factor: outdated artefact or stepping-stone to journal certification?," Scientometrics, Springer;Akadémiai Kiadó, vol. 92(2), pages 211-238, August.
    15. Jiancheng Guan & Ying He, 2007. "Patent-bibliometric analysis on the Chinese science — technology linkages," Scientometrics, Springer;Akadémiai Kiadó, vol. 72(3), pages 403-425, September.
    16. Guijie Zhang & Yuqiang Feng & Guang Yu & Luning Liu & Yanqiqi Hao, 2017. "Analyzing the time delay between scientific research and technology patents based on the citation distribution model," Scientometrics, Springer;Akadémiai Kiadó, vol. 111(3), pages 1287-1306, June.
    17. Chul Lee & Gunno Park & Klaus Marhold & Jina Kang, 2017. "Top management team’s innovation-related characteristics and the firm’s explorative R&D: an analysis based on patent data," Scientometrics, Springer;Akadémiai Kiadó, vol. 111(2), pages 639-663, May.
    18. Meyer, Martin, 2006. "Are patenting scientists the better scholars?: An exploratory comparison of inventor-authors with their non-inventing peers in nano-science and technology," Research Policy, Elsevier, vol. 35(10), pages 1646-1662, December.
    19. Sung, Hui-Yun & Wang, Chun-Chieh & Huang, Mu-Hsuan & Chen, Dar-Zen, 2015. "Measuring science-based science linkage and non-science-based linkage of patents through non-patent references," Journal of Informetrics, Elsevier, vol. 9(3), pages 488-498.
    20. Joaquín M. Azagra-Caro & Ignacio Fernández-de-Lucio & François Perruchas & Pauline Mattsson, 2009. "What do patent examiner inserted citations indicate for a region with low absorptive capacity?," Scientometrics, Springer;Akadémiai Kiadó, vol. 80(2), pages 441-455, August.

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:spr:scient:v:101:y:2014:i:1:d:10.1007_s11192-014-1337-0. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Sonal Shukla or Springer Nature Abstracting and Indexing (email available below). General contact details of provider: http://www.springer.com .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.