IDEAS home Printed from https://ideas.repec.org/a/spr/scient/v97y2013i2d10.1007_s11192-013-0993-9.html
   My bibliography  Save this article

Assessing non-standard article impact using F1000 labels

Author

Listed:
  • Ehsan Mohammadi

    (University of Wolverhampton)

  • Mike Thelwall

    (University of Wolverhampton)

Abstract

Faculty of 1000 (F1000) is a post-publishing peer review web site where experts evaluate and rate biomedical publications. F1000 reviewers also assign labels to each paper from a standard list or article types. This research examines the relationship between article types, citation counts and F1000 article factors (FFa). For this purpose, a random sample of F1000 medical articles from the years 2007 and 2008 were studied. In seven out of the nine cases, there were no significant differences between the article types in terms of citation counts and FFa scores. Nevertheless, citation counts and FFa scores were significantly different for two article types: “New finding” and “Changes clinical practice”: FFa scores value the appropriateness of medical research for clinical practice and “New finding” articles are more highly cited. It seems that highlighting key features of medical articles alongside ratings by Faculty members of F1000 could help to reveal the hidden value of some medical papers.

Suggested Citation

  • Ehsan Mohammadi & Mike Thelwall, 2013. "Assessing non-standard article impact using F1000 labels," Scientometrics, Springer;Akadémiai Kiadó, vol. 97(2), pages 383-395, November.
    • Handle: RePEc:spr:scient:v:97:y:2013:i:2:d:10.1007_s11192-013-0993-9
    DOI: 10.1007/s11192-013-0993-9
    as

    Download full text from publisher

    File URL: http://link.springer.com/10.1007/s11192-013-0993-9
    File Function: Abstract
    Download Restriction: Access to the full text of the articles in this series is restricted.
    File URL: https://libkey.io/10.1007/s11192-013-0993-9?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    As the access to this document is restricted, you may want to search for a different version of it.

    References listed on IDEAS

    as
    1. Liwen Vaughan & Debora Shaw, 2005. "Web citation data for impact assessment: A comparison of four science disciplines," Journal of the American Society for Information Science and Technology, Association for Information Science & Technology, vol. 56(10), pages 1075-1087, August.
    2. M.H. MacRoberts & B.R. MacRoberts, 2010. "Problems of citation analysis: A study of uncited and seldom-cited influences," Journal of the Association for Information Science & Technology, Association for Information Science & Technology, vol. 61(1), pages 1-12, January.
    3. Steve Hanney & Iain Frame & Jonathan Grant & Martin Buxton & Tracey Young & Grant Lewison, 2005. "Using categorisations of citations when assessing the outcomes from health research," Scientometrics, Springer;Akadémiai Kiadó, vol. 65(3), pages 357-379, December.
    4. Ronald N. Kostoff, 2007. "The difference between highly and poorly cited medical articles in the journal Lancet," Scientometrics, Springer;Akadémiai Kiadó, vol. 72(3), pages 513-520, September.
    5. Kayvan Kousha & Mike Thelwall & Somayeh Rezaie, 2011. "Assessing the citation impact of books: The role of Google Books, Google Scholar, and Scopus," Journal of the Association for Information Science & Technology, Association for Information Science & Technology, vol. 62(11), pages 2147-2164, November.
    6. Éric Archambault & David Campbell & Yves Gingras & Vincent Larivière, 2009. "Comparing bibliometric statistics obtained from the Web of Science and Scopus," Journal of the American Society for Information Science and Technology, Association for Information Science & Technology, vol. 60(7), pages 1320-1326, July.
    7. Richard E. Stern, 1990. "Uncitedness in the biomedical literature," Journal of the American Society for Information Science, Association for Information Science & Technology, vol. 41(3), pages 193-196, April.
    8. María‐del‐Mar Camacho‐Miñano & Manuel Núñez‐Nickel, 2009. "The multilayered nature of reference selection," Journal of the American Society for Information Science and Technology, Association for Information Science & Technology, vol. 60(4), pages 754-777, April.
    9. Anthony F. J. Raan, 2006. "Comparison of the Hirsch-index with standard bibliometric indicators and with peer judgment for 147 chemistry research groups," Scientometrics, Springer;Akadémiai Kiadó, vol. 67(3), pages 491-502, June.
    10. Emanuela Reale & Anna Barbara & Antonio Costantini, 2007. "Peer review for the evaluation of academic research: lessons from the Italian experience," Research Evaluation, Oxford University Press, vol. 16(3), pages 216-228, September.
    11. Teresa H. Jones & Claire Donovan & Steve Hanney, 2012. "Tracing the wider impacts of biomedical research: a literature search to develop a novel citation categorisation technique," Scientometrics, Springer;Akadémiai Kiadó, vol. 93(1), pages 125-134, October.
    12. Gunther Maier, 2006. "Impact factors and peer judgment: The case of regional science journals," Scientometrics, Springer;Akadémiai Kiadó, vol. 69(3), pages 651-667, December.
    13. Kayvan Kousha & Mike Thelwall, 2008. "Assessing the impact of disciplinary research on teaching: An automatic analysis of online syllabuses," Journal of the American Society for Information Science and Technology, Association for Information Science & Technology, vol. 59(13), pages 2060-2069, November.
    14. Nederhof, A. J. & van Raan, A. F. J., 1993. "A bibliometric analysis of six economics research groups: A comparison with peer review," Research Policy, Elsevier, vol. 22(4), pages 353-368, August.
    15. Tobias Opthof & Loet Leydesdorff, 2011. "A comment to the paper by Waltman et al., Scientometrics, 87, 467–481, 2011," Scientometrics, Springer;Akadémiai Kiadó, vol. 88(3), pages 1011-1016, September.
    16. M.H. MacRoberts & B.R. MacRoberts, 2010. "Problems of citation analysis: A study of uncited and seldom‐cited influences," Journal of the American Society for Information Science and Technology, Association for Information Science & Technology, vol. 61(1), pages 1-12, January.
    17. Heather A Piwowar & Roger S Day & Douglas B Fridsma, 2007. "Sharing Detailed Research Data Is Associated with Increased Citation Rate," PLOS ONE, Public Library of Science, vol. 2(3), pages 1-5, March.
    18. Ludo Waltman & Nees Jan Eck & Thed N. Leeuwen & Martijn S. Visser & Anthony F. J. Raan, 2011. "On the correlation between bibliometric indicators and peer review: reply to Opthof and Leydesdorff," Scientometrics, Springer;Akadémiai Kiadó, vol. 88(3), pages 1017-1022, September.
    19. Alesia Zuccala, 2010. "The mathematical review system: does reviewer status play a role in the citation process?," Scientometrics, Springer;Akadémiai Kiadó, vol. 84(1), pages 221-235, July.
    20. Franceschet, Massimo & Costantini, Antonio, 2011. "The first Italian research assessment exercise: A bibliometric perspective," Journal of Informetrics, Elsevier, vol. 5(2), pages 275-291.
    21. Kayvan Kousha & Mike Thelwall & Somayeh Rezaie, 2011. "Assessing the citation impact of books: The role of Google Books, Google Scholar, and Scopus," Journal of the American Society for Information Science and Technology, Association for Information Science & Technology, vol. 62(11), pages 2147-2164, November.
    22. Dag W Aksnes & Randi Elisabeth Taxt, 2004. "Peer reviews and bibliometric indicators: a comparative study at a Norwegian university," Research Evaluation, Oxford University Press, vol. 13(1), pages 33-41, April.
    23. Price, Joseph & Simon, Kosali, 2009. "Patient education and the impact of new medical research," Journal of Health Economics, Elsevier, vol. 28(6), pages 1166-1174, December.
    24. Henry Small, 2004. "On the shoulders of Robert Merton: Towards a normative theory of citation," Scientometrics, Springer;Akadémiai Kiadó, vol. 60(1), pages 71-79, May.
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Thelwall, Mike & Fairclough, Ruth, 2015. "The influence of time and discipline on the magnitude of correlations between citation counts and quality scores," Journal of Informetrics, Elsevier, vol. 9(3), pages 529-541.
    2. Pardeep Sud & Mike Thelwall, 2014. "Evaluating altmetrics," Scientometrics, Springer;Akadémiai Kiadó, vol. 98(2), pages 1131-1143, February.
    3. Bornmann, Lutz, 2014. "Validity of altmetrics data for measuring societal impact: A study using data from Altmetric and F1000Prime," Journal of Informetrics, Elsevier, vol. 8(4), pages 935-950.
    4. Ehsan Mohammadi & Mike Thelwall & Stefanie Haustein & Vincent Larivière, 2015. "Who reads research articles? An altmetrics analysis of Mendeley user categories," Journal of the Association for Information Science & Technology, Association for Information Science & Technology, vol. 66(9), pages 1832-1846, September.
    5. Abdelghani Maddi & Aouatif de La Laurencie, 2018. "La dynamique des SHS françaises dans le Web of Science : un manque de représentativité ou de visibilité internationale ?," CEPN Working Papers hal-01922266, HAL.
    6. Liwei Zhang & Jue Wang, 2018. "Why highly cited articles are not highly tweeted? A biology case," Scientometrics, Springer;Akadémiai Kiadó, vol. 117(1), pages 495-509, October.
    7. Yu, Houqiang & Xu, Shenmeng & Xiao, Tingting & Hemminger, Brad M. & Yang, Siluo, 2017. "Global science discussed in local altmetrics: Weibo and its comparison with Twitter," Journal of Informetrics, Elsevier, vol. 11(2), pages 466-482.
    8. Bornmann, Lutz, 2014. "Do altmetrics point to the broader impact of research? An overview of benefits and disadvantages of altmetrics," Journal of Informetrics, Elsevier, vol. 8(4), pages 895-903.
    9. Wang, Peiling & Su, Jing, 2021. "Post-publication expert recommendations in faculty opinions (F1000Prime): Recommended articles and citations," Journal of Informetrics, Elsevier, vol. 15(3).
    10. Banshal, Sumit Kumar & Gupta, Solanki & Lathabai, Hiran H & Singh, Vivek Kumar, 2022. "Power Laws in altmetrics: An empirical analysis," Journal of Informetrics, Elsevier, vol. 16(3).
    11. Fei Shu & Wen Lou & Stefanie Haustein, 2018. "Can Twitter increase the visibility of Chinese publications?," Scientometrics, Springer;Akadémiai Kiadó, vol. 116(1), pages 505-519, July.
    12. Abdelghani Maddi & Aouatif De La Laurencie, 2018. "La dynamique des SHS françaises dans le Web of Science," CEPN Working Papers 2018-05, Centre d'Economie de l'Université de Paris Nord.
    13. Lutz Bornmann, 2015. "Interrater reliability and convergent validity of F1000Prime peer review," Journal of the Association for Information Science & Technology, Association for Information Science & Technology, vol. 66(12), pages 2415-2426, December.
    14. Mike Thelwall & Kayvan Kousha, 2016. "Are citations from clinical trials evidence of higher impact research? An analysis of ClinicalTrials.gov," Scientometrics, Springer;Akadémiai Kiadó, vol. 109(2), pages 1341-1351, November.
    15. Peiling Wang & Joshua Williams & Nan Zhang & Qiang Wu, 2020. "F1000Prime recommended articles and their citations: an exploratory study of four journals," Scientometrics, Springer;Akadémiai Kiadó, vol. 122(2), pages 933-955, February.
    16. Saeideh Ebrahimy & Jafar Mehrad & Fatemeh Setareh & Massoud Hosseinchari, 2016. "Path analysis of the relationship between visibility and citation: the mediating roles of save, discussion, and recommendation metrics," Scientometrics, Springer;Akadémiai Kiadó, vol. 109(3), pages 1497-1510, December.
    17. Bornmann, Lutz & Tekles, Alexander & Zhang, Helena H. & Ye, Fred Y., 2019. "Do we measure novelty when we analyze unusual combinations of cited references? A validation study of bibliometric novelty indicators based on F1000Prime data," Journal of Informetrics, Elsevier, vol. 13(4).
    18. Thelwall, Mike & Wilson, Paul, 2014. "Regression for citation data: An evaluation of different methods," Journal of Informetrics, Elsevier, vol. 8(4), pages 963-971.
    19. Mojisola Erdt & Aarthy Nagarajan & Sei-Ching Joanna Sin & Yin-Leng Theng, 2016. "Altmetrics: an analysis of the state-of-the-art in measuring research impact on social media," Scientometrics, Springer;Akadémiai Kiadó, vol. 109(2), pages 1117-1166, November.
    20. Fei Shu, 2017. "Comment to: Does China need to rethink its metrics- and citation-based research rewards policies?," Scientometrics, Springer;Akadémiai Kiadó, vol. 113(2), pages 1229-1231, November.

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Jacques Wainer & Paula Vieira, 2013. "Correlations between bibliometrics and peer evaluation for all disciplines: the evaluation of Brazilian scientists," Scientometrics, Springer;Akadémiai Kiadó, vol. 96(2), pages 395-410, August.
    2. Martin Ricker, 2017. "Letter to the Editor: About the quality and impact of scientific articles," Scientometrics, Springer;Akadémiai Kiadó, vol. 111(3), pages 1851-1855, June.
    3. Horbach, Serge & Aagaard, Kaare & Schneider, Jesper W., 2021. "Meta-Research: How problematic citing practices distort science," MetaArXiv aqyhg, Center for Open Science.
    4. Giovanni Abramo & Ciriaco Andrea D’Angelo & Emanuela Reale, 2019. "Peer review versus bibliometrics: Which method better predicts the scholarly impact of publications?," Scientometrics, Springer;Akadémiai Kiadó, vol. 121(1), pages 537-554, October.
    5. Iman Tahamtan & Askar Safipour Afshar & Khadijeh Ahamdzadeh, 2016. "Factors affecting number of citations: a comprehensive review of the literature," Scientometrics, Springer;Akadémiai Kiadó, vol. 107(3), pages 1195-1225, June.
    6. Vieira, Elizabeth S. & Cabral, José A.S. & Gomes, José A.N.F., 2014. "How good is a model based on bibliometric indicators in predicting the final decisions made by peers?," Journal of Informetrics, Elsevier, vol. 8(2), pages 390-405.
    7. Li, Jiang & Sanderson, Mark & Willett, Peter & Norris, Michael & Oppenheim, Charles, 2010. "Ranking of library and information science researchers: Comparison of data sources for correlating citation data, and expert judgments," Journal of Informetrics, Elsevier, vol. 4(4), pages 554-563.
    8. Gangan Prathap, 2012. "Energy indicators and percentile ranking normalization," Scientometrics, Springer;Akadémiai Kiadó, vol. 91(3), pages 997-1003, June.
    9. Thelwall, Mike & Fairclough, Ruth, 2015. "The influence of time and discipline on the magnitude of correlations between citation counts and quality scores," Journal of Informetrics, Elsevier, vol. 9(3), pages 529-541.
    10. Siluo Yang & Feng Ma & Yanhui Song & Junping Qiu, 2010. "A longitudinal analysis of citation distribution breadth for Chinese scholars," Scientometrics, Springer;Akadémiai Kiadó, vol. 85(3), pages 755-765, December.
    11. Zewen Hu & Yishan Wu & Jianjun Sun, 2018. "A quantitative analysis of determinants of non-citation using a panel data model," Scientometrics, Springer;Akadémiai Kiadó, vol. 116(2), pages 843-861, August.
    12. Tobias Opthof & Loet Leydesdorff, 2011. "A comment to the paper by Waltman et al., Scientometrics, 87, 467–481, 2011," Scientometrics, Springer;Akadémiai Kiadó, vol. 88(3), pages 1011-1016, September.
    13. Lutz Bornmann & Robin Haunschild, 2017. "Does evaluative scientometrics lose its main focus on scientific quality by the new orientation towards societal impact?," Scientometrics, Springer;Akadémiai Kiadó, vol. 110(2), pages 937-943, February.
    14. Bar-Ilan, Judit, 2008. "Informetrics at the beginning of the 21st century—A review," Journal of Informetrics, Elsevier, vol. 2(1), pages 1-52.
    15. Ashraf Maleki, 2022. "Why does library holding format really matter for book impact assessment?: Modelling the relationship between citations and altmetrics with print and electronic holdings," Scientometrics, Springer;Akadémiai Kiadó, vol. 127(2), pages 1129-1160, February.
    16. Bornmann, Lutz & Haunschild, Robin & Mutz, Rüdiger, 2020. "Should citations be field-normalized in evaluative bibliometrics? An empirical analysis based on propensity score matching," Journal of Informetrics, Elsevier, vol. 14(4).
    17. Thelwall, Mike, 2017. "Three practical field normalised alternative indicator formulae for research evaluation," Journal of Informetrics, Elsevier, vol. 11(1), pages 128-151.
    18. Daisuke Sakai, 2019. "Who is peer reviewed? Comparing publication patterns of peer-reviewed and non-peer-reviewed papers in Japanese political science," Scientometrics, Springer;Akadémiai Kiadó, vol. 121(1), pages 65-80, October.
    19. Abramo, Giovanni, 2018. "Revisiting the scientometric conceptualization of impact and its measurement," Journal of Informetrics, Elsevier, vol. 12(3), pages 590-597.
    20. Dag W. Aksnes & Liv Langfeldt & Paul Wouters, 2019. "Citations, Citation Indicators, and Research Quality: An Overview of Basic Concepts and Theories," SAGE Open, , vol. 9(1), pages 21582440198, February.

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:spr:scient:v:97:y:2013:i:2:d:10.1007_s11192-013-0993-9. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Sonal Shukla or Springer Nature Abstracting and Indexing (email available below). General contact details of provider: http://www.springer.com .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.