IDEAS home Printed from https://ideas.repec.org/a/spr/scient/v127y2022i12d10.1007_s11192-022-04327-4.html
   My bibliography  Save this article

Do papers (really) match journals’ “aims and scope”? A computational assessment of innovation studies

Author

Listed:
  • Ana Teresa Santos

    (Instituto Universitário de Lisboa (Iscte-IUL))

  • Sandro Mendonça

    (Instituto Universitário de Lisboa (Iscte-IUL)
    UECE/REM – ISEG/ University of Lisbon
    SPRU, University of Sussex)

Abstract

Researchers, science managers and evaluation professionals face a problem when determining the alignment between research results and publication targets. How does a manuscript’s content fit a given journal’s stated purpose? We develop a framework for understanding how past published papers reveal the actual interests and editorial profile of journal. We articulate an answer to the question by using a total of 16,803 abstracts from articles published from 2010 to 2019 in 20 top innovation-oriented journals. Through a machine learning approach, we trained a text classification algorithm on these materials. The supervised model matched the published contents (abstracts) with journal blurbs with an accuracy rate of 80%. We discover that the content of 25% of the outlet sample might have been of greater interest elsewhere (i.e. to other journals), according to the official editorial positioning available in their homepages. Our conclusions suggest that more can be learned from exploring the abstract-blurb nexus.

Suggested Citation

  • Ana Teresa Santos & Sandro Mendonça, 2022. "Do papers (really) match journals’ “aims and scope”? A computational assessment of innovation studies," Scientometrics, Springer;Akadémiai Kiadó, vol. 127(12), pages 7449-7470, December.
    • Handle: RePEc:spr:scient:v:127:y:2022:i:12:d:10.1007_s11192-022-04327-4
    DOI: 10.1007/s11192-022-04327-4
    as

    Download full text from publisher

    File URL: http://link.springer.com/10.1007/s11192-022-04327-4
    File Function: Abstract
    Download Restriction: Access to the full text of the articles in this series is restricted.
    File URL: https://libkey.io/10.1007/s11192-022-04327-4?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    As the access to this document is restricted, you may want to search for a different version of it.

    References listed on IDEAS

    as
    1. John P. A. Ioannidis & Richard Klavans & Kevin W. Boyack, 2018. "Thousands of scientists publish a paper every five days," Nature, Nature, vol. 561(7722), pages 167-169, September.
    2. Björk, Bo-Christer & Solomon, David, 2013. "The publishing delay in scholarly peer-reviewed journals," Journal of Informetrics, Elsevier, vol. 7(4), pages 914-923.
    3. Johan Bollen & Herbert Van de Sompel & Aric Hagberg & Luis Bettencourt & Ryan Chute & Marko A Rodriguez & Lyudmila Balakireva, 2009. "Clickstream Data Yields High-Resolution Maps of Science," PLOS ONE, Public Library of Science, vol. 4(3), pages 1-11, March.
    4. Sandro Mendonca & Hugo Confraria & Manuel Mira Godinho, 2021. "Appropriating the returns of patent statistics: Take-up and development in the wake of Zvi Griliches," SPRU Working Paper Series 2021-07, SPRU - Science Policy Research Unit, University of Sussex Business School.
    5. David J. Solomon & Bo‐Christer Björk, 2012. "Publication fees in open access publishing: Sources of funding and factors influencing choice of journal," Journal of the American Society for Information Science and Technology, Association for Information Science & Technology, vol. 63(1), pages 98-107, January.
    6. Martin, Ben R., 2012. "The evolution of science policy and innovation studies," Research Policy, Elsevier, vol. 41(7), pages 1219-1239.
    7. Fulvio Castellaci & Stine Grodal & Sandro Mendonca & Mona Wibe, 2005. "Advances and Challenges in Innovation Studies," Journal of Economic Issues, Taylor & Francis Journals, vol. 39(1), pages 91-121, March.
    8. Judit Bar-Ilan, 2008. "Which h-index? — A comparison of WoS, Scopus and Google Scholar," Scientometrics, Springer;Akadémiai Kiadó, vol. 74(2), pages 257-271, February.
    9. Johan Bollen & Herbert Van de Sompel & Aric Hagberg & Ryan Chute, 2009. "A Principal Component Analysis of 39 Scientific Impact Measures," PLOS ONE, Public Library of Science, vol. 4(6), pages 1-11, June.
    10. Martin, Ben R. & Irvine, John, 1993. "Assessing basic research : Some partial indicators of scientific progress in radio astronomy," Research Policy, Elsevier, vol. 22(2), pages 106-106, April.
    11. Fagerberg, Jan & Fosaas, Morten & Sapprasert, Koson, 2012. "Innovation: Exploring the knowledge base," Research Policy, Elsevier, vol. 41(7), pages 1132-1153.
    12. Lutz Bornmann & Rüdiger Mutz, 2015. "Growth rates of modern science: A bibliometric analysis based on the number of publications and cited references," Journal of the Association for Information Science & Technology, Association for Information Science & Technology, vol. 66(11), pages 2215-2222, November.
    13. Small, Henry & Boyack, Kevin W. & Klavans, Richard, 2014. "Identifying emerging topics in science and technology," Research Policy, Elsevier, vol. 43(8), pages 1450-1467.
    14. Vanclay, Jerome K., 2008. "Ranking forestry journals using the h-index," Journal of Informetrics, Elsevier, vol. 2(4), pages 326-334.
    15. Na Liu & Philip Shapira & Xiaoxu Yue, 2021. "Tracking developments in artificial intelligence research: constructing and applying a new search strategy," Scientometrics, Springer;Akadémiai Kiadó, vol. 126(4), pages 3153-3192, April.
    16. Titipat Achakulvisut & Daniel E Acuna & Tulakan Ruangrong & Konrad Kording, 2016. "Science Concierge: A Fast Content-Based Recommendation System for Scientific Publications," PLOS ONE, Public Library of Science, vol. 11(7), pages 1-11, July.
    17. Chen, P. & Xie, H. & Maslov, S. & Redner, S., 2007. "Finding scientific gems with Google’s PageRank algorithm," Journal of Informetrics, Elsevier, vol. 1(1), pages 8-15.
    18. Carolina Castaldi & Sandro Mendonça, 2022. "Regions and trademarks: research opportunities and policy insights from leveraging trademarks in regional innovation studies," Regional Studies, Taylor & Francis Journals, vol. 56(2), pages 177-189, February.
    19. Mark Hahnel, 2013. "Referencing: The reuse factor," Nature, Nature, vol. 502(7471), pages 298-298, October.
    20. Nadia Simoes & Nuno Crespo, 2020. "A flexible approach for measuring author-level publishing performance," Scientometrics, Springer;Akadémiai Kiadó, vol. 122(1), pages 331-355, January.
    21. John P. A. Ioannidis & Kevin W. Boyack & Henry Small & Aaron A. Sorensen & Richard Klavans, 2014. "Bibliometrics: Is your most cited work your best?," Nature, Nature, vol. 514(7524), pages 561-562, October.
    22. Manuel Ennes Ferreira & Sandro Mendonça & João Pereira, 2018. "Gatekeeping African studies: What does “editormetrics” indicate about journal governance?," Working Papers Department of Economics 2018/07, ISEG - Lisbon School of Economics and Management, Department of Economics, Universidade de Lisboa.
    23. Sandro Mendonça & João Pereira & Manuel Ennes Ferreira, 2018. "Gatekeeping African studies: what does “editormetrics” indicate about journal governance?," Scientometrics, Springer;Akadémiai Kiadó, vol. 117(3), pages 1513-1534, December.
    24. Castellacci, Fulvio & Grodal, Stine & Mendonca, Sandro & Wibe, Mona, 2005. "Advances and challenges in innovation studies," MPRA Paper 27519, University Library of Munich, Germany.
    25. Jaime A. Teixeira da Silva & Aamir Raoof Memon, 2017. "CiteScore: A cite for sore eyes, or a valuable, transparent metric?," Scientometrics, Springer;Akadémiai Kiadó, vol. 111(1), pages 553-556, April.
    26. Bornmann, Lutz & Marx, Werner, 2015. "Methods for the generation of normalized citation impact scores in bibliometrics: Which method best reflects the judgements of experts?," Journal of Informetrics, Elsevier, vol. 9(2), pages 408-418.
    27. Jan Fagerberg & Bart Verspagen, 2020. "Innovation–diffusion, the economy and contemporary challenges: a comment," Industrial and Corporate Change, Oxford University Press and the Associazione ICC, vol. 29(4), pages 1067-1073.
    28. Zeynep Hilal Kilimci & A. Okay Akyuz & Mitat Uysal & Selim Akyokus & M. Ozan Uysal & Berna Atak Bulbul & Mehmet Ali Ekmis, 2019. "An Improved Demand Forecasting Model Using Deep Learning Approach and Proposed Decision Integration Strategy for Supply Chain," Complexity, Hindawi, vol. 2019, pages 1-15, March.
    29. Hugo Confraria & Manuel Mira Godinho, 2015. "The impact of African science: a bibliometric analysis," Scientometrics, Springer;Akadémiai Kiadó, vol. 102(2), pages 1241-1268, February.
    30. Sandra Rousseau & Ronald Rousseau, 2021. "Bibliometric Techniques And Their Use In Business And Economics Research," Journal of Economic Surveys, Wiley Blackwell, vol. 35(5), pages 1428-1451, December.
    31. Lutz Bornmann & Hans‐Dieter Daniel, 2007. "What do we know about the h index?," Journal of the American Society for Information Science and Technology, Association for Information Science & Technology, vol. 58(9), pages 1381-1385, July.
    32. Ana Teresa Santos & Sandro Mendonça, 2022. "The small world of innovation studies: an “editormetrics” perspective," Scientometrics, Springer;Akadémiai Kiadó, vol. 127(12), pages 7471-7486, December.
    33. James Hartley & Guillaume Cabanac, 2017. "Thirteen Ways to Write an Abstract," Publications, MDPI, vol. 5(2), pages 1-7, May.
    34. Moed, Henk F., 2010. "Measuring contextual citation impact of scientific journals," Journal of Informetrics, Elsevier, vol. 4(3), pages 265-277.
    35. Michail Kovanis & Raphaël Porcher & Philippe Ravaud & Ludovic Trinquart, 2016. "The Global Burden of Journal Peer Review in the Biomedical Literature: Strong Imbalance in the Collective Enterprise," PLOS ONE, Public Library of Science, vol. 11(11), pages 1-14, November.
    36. David J. Solomon & Bo-Christer Björk, 2012. "Publication fees in open access publishing: Sources of funding and factors influencing choice of journal," Journal of the Association for Information Science & Technology, Association for Information Science & Technology, vol. 63(1), pages 98-107, January.
    37. Diana Hicks & Paul Wouters & Ludo Waltman & Sarah de Rijcke & Ismael Rafols, 2015. "Bibliometrics: The Leiden Manifesto for research metrics," Nature, Nature, vol. 520(7548), pages 429-431, April.
    38. Alexandre Paredes & Joana Mendonça & Fernando Bação & Bruno Damásio, 2022. "Does R&D tax credit impact firm behaviour? Micro evidence for Portugal," Research Evaluation, Oxford University Press, vol. 31(2), pages 226-235.
    39. Xin Gu & Karen L. Blackmore, 2016. "Recent trends in academic journal growth," Scientometrics, Springer;Akadémiai Kiadó, vol. 108(2), pages 693-716, August.
    Full references (including those not matched with items on IDEAS)

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Ana Teresa Santos & Sandro Mendonça, 2022. "The small world of innovation studies: an “editormetrics” perspective," Scientometrics, Springer;Akadémiai Kiadó, vol. 127(12), pages 7471-7486, December.
    2. Raminta Pranckutė, 2021. "Web of Science (WoS) and Scopus: The Titans of Bibliographic Information in Today’s Academic World," Publications, MDPI, vol. 9(1), pages 1-59, March.
    3. B Ian Hutchins & Xin Yuan & James M Anderson & George M Santangelo, 2016. "Relative Citation Ratio (RCR): A New Metric That Uses Citation Rates to Measure Influence at the Article Level," PLOS Biology, Public Library of Science, vol. 14(9), pages 1-25, September.
    4. Leckel, Anja & Veilleux, Sophie & Dana, Leo Paul, 2020. "Local Open Innovation: A means for public policy to increase collaboration for innovation in SMEs," Technological Forecasting and Social Change, Elsevier, vol. 153(C).
    5. Xie, Qing & Zhang, Xinyuan & Song, Min, 2021. "A network embedding-based scholar assessment indicator considering four facets: Research topic, author credit allocation, field-normalized journal impact, and published time," Journal of Informetrics, Elsevier, vol. 15(4).
    6. Waltman, Ludo, 2016. "A review of the literature on citation impact indicators," Journal of Informetrics, Elsevier, vol. 10(2), pages 365-391.
    7. Callegari, Beniamino & Nybakk, Erlend, 2022. "Schumpeterian theory and research on forestry innovation and entrepreneurship: The state of the art, issues and an agenda," Forest Policy and Economics, Elsevier, vol. 138(C).
    8. Metzger, Philipp & Mendonça, Sandro & Silva, José A. & Damásio, Bruno, 2023. "Battery innovation and the Circular Economy: What are patents revealing?," Renewable Energy, Elsevier, vol. 209(C), pages 516-532.
    9. Vanclay, Jerome K., 2011. "An evaluation of the Australian Research Council's journal ranking," Journal of Informetrics, Elsevier, vol. 5(2), pages 265-274.
    10. Walters, William H., 2017. "Do subjective journal ratings represent whole journals or typical articles? Unweighted or weighted citation impact?," Journal of Informetrics, Elsevier, vol. 11(3), pages 730-744.
    11. Dunaiski, Marcel & Geldenhuys, Jaco & Visser, Willem, 2019. "On the interplay between normalisation, bias, and performance of paper impact metrics," Journal of Informetrics, Elsevier, vol. 13(1), pages 270-290.
    12. Mendonça, Sandro & Damásio, Bruno & Charlita de Freitas, Luciano & Oliveira, Luís & Cichy, Marcin & Nicita, António, 2022. "The rise of 5G technologies and systems: A quantitative analysis of knowledge production," Telecommunications Policy, Elsevier, vol. 46(4).
    13. Yu Zhang & Min Wang & Morteza Saberi & Elizabeth Chang, 2022. "Analysing academic paper ranking algorithms using test data and benchmarks: an investigation," Scientometrics, Springer;Akadémiai Kiadó, vol. 127(7), pages 4045-4074, July.
    14. Mingkun Wei, 2020. "Research on impact evaluation of open access journals," Scientometrics, Springer;Akadémiai Kiadó, vol. 122(2), pages 1027-1049, February.
    15. Mingers, John & Yang, Liying, 2017. "Evaluating journal quality: A review of journal citation indicators and ranking in business and management," European Journal of Operational Research, Elsevier, vol. 257(1), pages 323-337.
    16. Cameron Neylon & Shirley Wu, 2009. "Article-Level Metrics and the Evolution of Scientific Impact," PLOS Biology, Public Library of Science, vol. 7(11), pages 1-6, November.
    17. Peter Jacso, 2012. "Grim tales about the impact factor and the h-index in the Web of Science and the Journal Citation Reports databases: reflections on Vanclay’s criticism," Scientometrics, Springer;Akadémiai Kiadó, vol. 92(2), pages 325-354, August.
    18. Yuya Kajikawa, 2022. "Reframing evidence in evidence-based policy making and role of bibliometrics: toward transdisciplinary scientometric research," Scientometrics, Springer;Akadémiai Kiadó, vol. 127(9), pages 5571-5585, September.
    19. Paul Sebo & Jean Pascal Fournier & Claire Ragot & Pierre-Henri Gorioux & François R. Herrmann & Hubert Maisonneuve, 2019. "Factors associated with publication speed in general medical journals: a retrospective study of bibliometric data," Scientometrics, Springer;Akadémiai Kiadó, vol. 119(2), pages 1037-1058, May.
    20. Lutz Bornmann & Alexander Tekles & Loet Leydesdorff, 2019. "How well does I3 perform for impact measurement compared to other bibliometric indicators? The convergent validity of several (field-normalized) indicators," Scientometrics, Springer;Akadémiai Kiadó, vol. 119(2), pages 1187-1205, May.

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:spr:scient:v:127:y:2022:i:12:d:10.1007_s11192-022-04327-4. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Sonal Shukla or Springer Nature Abstracting and Indexing (email available below). General contact details of provider: http://www.springer.com .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.