IDEAS home Printed from https://ideas.repec.org/a/bla/jinfst/v72y2021i6p777-792.html
   My bibliography  Save this article

Conjoint analysis of researchers' hidden preferences for bibliometrics, altmetrics, and usage metrics

Author

Listed:
  • Steffen Lemke
  • Athanasios Mazarakis
  • Isabella Peters

Abstract

The amount of annually published scholarly articles is growing steadily, as is the number of indicators through which impact of publications is measured. Little is known about how the increasing variety of available metrics affects researchers' processes of selecting literature to read. We conducted ranking experiments embedded into an online survey with 247 participating researchers, most from social sciences. Participants completed series of tasks in which they were asked to rank fictitious publications regarding their expected relevance, based on their scores regarding six prototypical metrics. Through applying logistic regression, cluster analysis, and manual coding of survey answers, we obtained detailed data on how prominent metrics for research impact influence our participants in decisions about which scientific articles to read. Survey answers revealed a combination of qualitative and quantitative characteristics that researchers consult when selecting literature, while regression analysis showed that among quantitative metrics, citation counts tend to be of highest concern, followed by Journal Impact Factors. Our results suggest a comparatively favorable view of many researchers on bibliometrics and widespread skepticism toward altmetrics. The findings underline the importance of equipping researchers with solid knowledge about specific metrics' limitations, as they seem to play significant roles in researchers' everyday relevance assessments.

Suggested Citation

  • Steffen Lemke & Athanasios Mazarakis & Isabella Peters, 2021. "Conjoint analysis of researchers' hidden preferences for bibliometrics, altmetrics, and usage metrics," Journal of the Association for Information Science & Technology, Association for Information Science & Technology, vol. 72(6), pages 777-792, June.
    • Handle: RePEc:bla:jinfst:v:72:y:2021:i:6:p:777-792
    DOI: 10.1002/asi.24445
    as

    Download full text from publisher

    File URL: https://doi.org/10.1002/asi.24445
    Download Restriction: no
    File URL: https://libkey.io/10.1002/asi.24445?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    References listed on IDEAS

    as
    1. Björn Hammarfelt & Gaby Haddow, 2018. "Conflicting measures and values: How humanities scholars in Australia and Sweden use and react to bibliometric indicators," Journal of the Association for Information Science & Technology, Association for Information Science & Technology, vol. 69(7), pages 924-935, July.
    2. Carol Tenopir & Kenneth Levine & Suzie Allard & Lisa Christian & Rachel Volentine & Reid Boehm & Frances Nichols & David Nicholas & Hamid R. Jamali & Eti Herman & Anthony Watkinson, 2016. "Trustworthiness and authority of scholarly information in a digital age: Results of an international questionnaire," Journal of the Association for Information Science & Technology, Association for Information Science & Technology, vol. 67(10), pages 2344-2361, October.
    3. Alison Abbott & David Cyranoski & Nicola Jones & Brendan Maher & Quirin Schiermeier & Richard Van Noorden, 2010. "Metrics: Do metrics matter?," Nature, Nature, vol. 465(7300), pages 860-862, June.
    4. Xi Niu & Bradley M. Hemminger, 2012. "A study of factors that affect the information-seeking behavior of academic scientists," Journal of the Association for Information Science & Technology, Association for Information Science & Technology, vol. 63(2), pages 336-353, February.
    5. Gunnar Sivertsen & Birger Larsen, 2012. "Comprehensive bibliographic coverage of the social sciences and humanities in a citation index: an empirical analysis of the potential," Scientometrics, Springer;Akadémiai Kiadó, vol. 91(2), pages 567-575, May.
    6. Diana Hicks, 2005. "The Four Literatures Of Social Sciences," IBT Journal of Business Studies (JBS), Ilma University, Faculty of Management Science, vol. 1(1), pages 1-1.
    7. Lutz Bornmann & Rüdiger Mutz, 2015. "Growth rates of modern science: A bibliometric analysis based on the number of publications and cited references," Journal of the Association for Information Science & Technology, Association for Information Science & Technology, vol. 66(11), pages 2215-2222, November.
    8. Aksnes, Dag W. & Rip, Arie, 2009. "Researchers' perceptions of citations," Research Policy, Elsevier, vol. 38(6), pages 895-905, July.
    9. Linda Butler, 2007. "Assessing university research: A plea for a balanced approach," Science and Public Policy, Oxford University Press, vol. 34(8), pages 565-574, October.
    10. Stefanie Haustein & Isabella Peters & Judit Bar-Ilan & Jason Priem & Hadas Shema & Jens Terliesner, 2014. "Coverage and adoption of altmetrics sources in the bibliometric community," Scientometrics, Springer;Akadémiai Kiadó, vol. 101(2), pages 1145-1163, November.
    11. Diana Hicks, 2005. "The Four Literatures Of Social Sciences," IBT Journal of Business Studies (JBS), Ilma University, Faculty of Management Science, vol. 1(1), pages 1-20.
    12. Michael H. MacRoberts & Barbara R. MacRoberts, 2018. "The mismeasure of science: Citation analysis," Journal of the Association for Information Science & Technology, Association for Information Science & Technology, vol. 69(3), pages 474-482, March.
    13. David Nicholas & Eti Herman & Hamid R Jamali & Abdullah Abrizah & Cherifa Boukacem-Zeghmouri & Jie Xu & Blanca Rodríguez-Bravo & Anthony Watkinson & Tatiana Polezhaeva & Marzena Świgon, 2020. "Millennial researchers in a metric-driven scholarly world: An international study," Research Evaluation, Oxford University Press, vol. 29(3), pages 263-274.
    14. Xi Niu & Bradley M. Hemminger, 2012. "A study of factors that affect the information‐seeking behavior of academic scientists," Journal of the American Society for Information Science and Technology, Association for Information Science & Technology, vol. 63(2), pages 336-353, February.
    15. Htet Htet Aung & Han Zheng & Mojisola Erdt & Ashley Sara Aw & Sei‐Ching Joanna Sin & Yin‐Leng Theng, 2019. "Investigating familiarity and usage of traditional metrics and altmetrics," Journal of the Association for Information Science & Technology, Association for Information Science & Technology, vol. 70(8), pages 872-887, August.
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Tutku Tuncalı Yaman & Özgür Çakır, 2021. "A Game-Theoretical Approach to Conjoint Analysis," Alphanumeric Journal, Bahadir Fatih Yildirim, vol. 9(2), pages 179-216, December.
    2. Mike Thelwall & Kayvan Kousha, 2021. "Researchers’ attitudes towards the h-index on Twitter 2007–2020: criticism and acceptance," Scientometrics, Springer;Akadémiai Kiadó, vol. 126(6), pages 5361-5368, June.
    3. Ramón A. Feenstra & Emilio Delgado López-Cózar, 2022. "Philosophers’ appraisals of bibliometric indicators and their use in evaluation: from recognition to knee-jerk rejection," Scientometrics, Springer;Akadémiai Kiadó, vol. 127(4), pages 2085-2103, April.

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Ramón A. Feenstra & Emilio Delgado López-Cózar, 2022. "Philosophers’ appraisals of bibliometric indicators and their use in evaluation: from recognition to knee-jerk rejection," Scientometrics, Springer;Akadémiai Kiadó, vol. 127(4), pages 2085-2103, April.
    2. Liang, Guoqiang & Hou, Haiyan & Ding, Ying & Hu, Zhigang, 2020. "Knowledge recency to the birth of Nobel Prize-winning articles: Gender, career stage, and country," Journal of Informetrics, Elsevier, vol. 14(3).
    3. Pantea Kamrani & Isabelle Dorsch & Wolfgang G. Stock, 2021. "Do researchers know what the h-index is? And how do they estimate its importance?," Scientometrics, Springer;Akadémiai Kiadó, vol. 126(7), pages 5489-5508, July.
    4. Galt, Ryan E. & Pinzón, Natalia & Robinson, Nicholas Ian & Baukloh Coronil, Marcela Beatriz, 2024. "Agroecology and the social sciences: A half-century systematic review," Agricultural Systems, Elsevier, vol. 216(C).
    5. Alice Fleerackers & Lise Nehring & Lauren A. Maggio & Asura Enkhbayar & Laura Moorhead & Juan Pablo Alperin, 2022. "Identifying science in the news: An assessment of the precision and recall of Altmetric.com news mention data," Scientometrics, Springer;Akadémiai Kiadó, vol. 127(11), pages 6109-6123, November.
    6. Jinseok Kim, 2018. "Evaluating author name disambiguation for digital libraries: a case of DBLP," Scientometrics, Springer;Akadémiai Kiadó, vol. 116(3), pages 1867-1886, September.
    7. Mahmood Khosrowjerdi & Anneli Sundqvist & Katriina Byström, 2020. "Cultural Patterns of Information Source Use: A Global Study of 47 Countries," Journal of the Association for Information Science & Technology, Association for Information Science & Technology, vol. 71(6), pages 711-724, June.
    8. Elizabeth Martín-Mora & Shari Ellis & Lawrence M Page, 2020. "Use of web-based species occurrence information systems by academics and government professionals," PLOS ONE, Public Library of Science, vol. 15(7), pages 1-41, July.
    9. Qingzhou Luo & Jianhong Cecilia Xia & Gaby Haddow & Michele Willson & Jun Yang, 2018. "Does distance hinder the collaboration between Australian universities in the humanities, arts and social sciences?," Scientometrics, Springer;Akadémiai Kiadó, vol. 115(2), pages 695-715, May.
    10. Mike Thelwall & Kayvan Kousha & Mahshid Abdoli & Emma Stuart & Meiko Makita & Paul Wilson & Jonathan Levitt, 2023. "Do altmetric scores reflect article quality? Evidence from the UK Research Excellence Framework 2021," Journal of the Association for Information Science & Technology, Association for Information Science & Technology, vol. 74(5), pages 582-593, May.
    11. Claudia N. González Brambila & José Luis Olivares-Vázquez, 2021. "Patterns and evolution of publication and co-authorship in Social Sciences in Mexico," Scientometrics, Springer;Akadémiai Kiadó, vol. 126(3), pages 2595-2626, March.
    12. Ehsan Mohammadi & Mike Thelwall & Stefanie Haustein & Vincent Larivière, 2015. "Who reads research articles? An altmetrics analysis of Mendeley user categories," Journal of the Association for Information Science & Technology, Association for Information Science & Technology, vol. 66(9), pages 1832-1846, September.
    13. Nestor Gandelman & Osiris J. Parcero & Matilde Pereira & Flavia Roldán, 2021. "Ventajas comparativas reveladas en disciplinas científicas y tecnológicas en Uruguay," Documentos de Investigación 125, Universidad ORT Uruguay. Facultad de Administración y Ciencias Sociales.
    14. Sergey Kolesnikov & Eriko Fukumoto & Barry Bozeman, 2018. "Researchers’ risk-smoothing publication strategies: Is productivity the enemy of impact?," Scientometrics, Springer;Akadémiai Kiadó, vol. 116(3), pages 1995-2017, September.
    15. Verleysen, Frederik T. & Engels, Tim C.E., 2014. "Barycenter representation of book publishing internationalization in the Social Sciences and Humanities," Journal of Informetrics, Elsevier, vol. 8(1), pages 234-240.
    16. Kaur, Jasleen & Radicchi, Filippo & Menczer, Filippo, 2013. "Universality of scholarly impact metrics," Journal of Informetrics, Elsevier, vol. 7(4), pages 924-932.
    17. Ramona Weinrich, 2019. "Opportunities for the Adoption of Health-Based Sustainable Dietary Patterns: A Review on Consumer Research of Meat Substitutes," Sustainability, MDPI, vol. 11(15), pages 1-15, July.
    18. Piers Steel & Sjoerd Beugelsdijk & Herman Aguinis, 2021. "The anatomy of an award-winning meta-analysis: Recommendations for authors, reviewers, and readers of meta-analytic reviews," Journal of International Business Studies, Palgrave Macmillan;Academy of International Business, vol. 52(1), pages 23-44, February.
    19. Dunaiski, Marcel & Geldenhuys, Jaco & Visser, Willem, 2019. "On the interplay between normalisation, bias, and performance of paper impact metrics," Journal of Informetrics, Elsevier, vol. 13(1), pages 270-290.
    20. Augusteijn, Hilde Elisabeth Maria & van Aert, Robbie Cornelis Maria & van Assen, Marcel A. L. M., 2021. "Posterior Probabilities of Effect Sizes and Heterogeneity in Meta-Analysis: An Intuitive Approach of Dealing with Publication Bias," OSF Preprints avkgj, Center for Open Science.

    More about this item

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:bla:jinfst:v:72:y:2021:i:6:p:777-792. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Wiley Content Delivery (email available below). General contact details of provider: http://www.asis.org .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.