IDEAS home Printed from https://ideas.repec.org/a/eee/respol/v52y2023i5s0048733323000379.html
   My bibliography  Save this article

Ranking researchers: Evidence from Indonesia

Author

Listed:
  • Fry, Caroline V.
  • Lynham, John
  • Tran, Shannon

Abstract

In the span of three years, Indonesia went from being the second worst to the top producer of scientific journal articles in Southeast Asia. We investigate whether a transparent system of ranking every single researcher in the country based on publications and citations (SINTA) contributed to this turnaround. Using panel data from over 200,000 Indonesian researchers (and comparing to researchers from Thailand and the Philippines), we show that the implementation of SINTA coincides with changes in the production of publications by Indonesian researchers consistent with the weights used in the ranking formula. Although we see modest improvements in publication rates in top-ranked journals, 62 % of the observed increase in total publications is from conference proceedings. Because SINTA was launched around the same time as other policies that focused on increasing publications, isolating the precise impact of SINTA remains challenging. Nevertheless, after accounting for such policies, our results imply that a ranking and evaluation system for researchers can contribute to overall improvements in scientific capacity in low- and middle-income countries.

Suggested Citation

  • Fry, Caroline V. & Lynham, John & Tran, Shannon, 2023. "Ranking researchers: Evidence from Indonesia," Research Policy, Elsevier, vol. 52(5).
    • Handle: RePEc:eee:respol:v:52:y:2023:i:5:s0048733323000379
    DOI: 10.1016/j.respol.2023.104753
    as

    Download full text from publisher

    File URL: http://www.sciencedirect.com/science/article/pii/S0048733323000379
    Download Restriction: Full text for ScienceDirect subscribers only
    File URL: https://libkey.io/10.1016/j.respol.2023.104753?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    As the access to this document is restricted, you may want to search for a different version of it.

    References listed on IDEAS

    as
    1. Ina Ganguli, 2017. "Saving Soviet Science: The Impact of Grants When Government R&D Funding Disappears," American Economic Journal: Applied Economics, American Economic Association, vol. 9(2), pages 165-201, April.
    2. Pierre Azoulay & Joshua S. Graff Zivin & Gustavo Manso, 2011. "Incentives and creativity: evidence from the academic life sciences," RAND Journal of Economics, RAND Corporation, vol. 42(3), pages 527-554, September.
    3. David Gill & Zdenka Kissová & Jaesun Lee & Victoria Prowse, 2019. "First-Place Loving and Last-Place Loathing: How Rank in the Distribution of Performance Affects Effort Provision," Management Science, INFORMS, vol. 65(2), pages 494-507, February.
    4. Jeffrey T. Denning & Richard Murphy & Felix Weinhardt, 2023. "Class Rank and Long-Run Outcomes," The Review of Economics and Statistics, MIT Press, vol. 105(6), pages 1426-1441, November.
    5. David Card & Alexandre Mas & Enrico Moretti & Emmanuel Saez, 2012. "Inequality at Work: The Effect of Peer Salaries on Job Satisfaction," American Economic Review, American Economic Association, vol. 102(6), pages 2981-3003, October.
    6. Esther Duflo & Rema Hanna & Stephen P. Ryan, 2012. "Incentives Work: Getting Teachers to Come to School," American Economic Review, American Economic Association, vol. 102(4), pages 1241-1278, June.
    7. Seeber, Marco & Cattaneo, Mattia & Meoli, Michele & Malighetti, Paolo, 2019. "Self-citations as strategic response to the use of metrics for career decisions," Research Policy, Elsevier, vol. 48(2), pages 478-491.
    8. Federico Scarpa & Vincenzo Bianco & Luca A. Tagliafico, 2018. "The impact of the national assessment exercises on self-citation rate and publication venue: an empirical investigation on the engineering academic sector in Italy," Scientometrics, Springer;Akadémiai Kiadó, vol. 117(2), pages 997-1022, November.
    9. Michael Kosfeld & Susanne Neckermann, 2011. "Getting More Work for Nothing? Symbolic Awards and Worker Performance," American Economic Journal: Microeconomics, American Economic Association, vol. 3(3), pages 86-99, August.
    10. Ashish Arora & Alfonso Gambardella, 2010. "The Impact of NSF Support for Basic Research in Economics," NBER Chapters, in: Contributions in Memory of Zvi Griliches, pages 91-115, National Bureau of Economic Research, Inc.
    11. Richard Murphy & Felix Weinhardt, 2020. "Top of the Class: The Importance of Ordinal Rank," The Review of Economic Studies, Review of Economic Studies Ltd, vol. 87(6), pages 2777-2826.
    12. Ruchir Agarwal & Patrick Gaule, 2020. "Invisible Geniuses: Could the Knowledge Frontier Advance Faster?," American Economic Review: Insights, American Economic Association, vol. 2(4), pages 409-424, December.
    13. Tran, Anh & Zeckhauser, Richard, 2012. "Rank as an inherent incentive: Evidence from a field experiment," Journal of Public Economics, Elsevier, vol. 96(9-10), pages 645-650.
    14. Aditya Alta, 2020. "Beyond The Scopus Frenzy: Policy Alternatives To Incentivize Academic Publications," LPEM FEBUI Working Papers 202051, LPEM, Faculty of Economics and Business, University of Indonesia, revised 2020.
    15. Christiane Bradler & Robert Dur & Susanne Neckermann & Arjan Non, 2016. "Employee Recognition and Performance: A Field Experiment," Management Science, INFORMS, vol. 62(11), pages 3085-3099, November.
    16. Shulamit Kahn & Megan J. MacGarvie, 2016. "How Important Is U.S. Location for Research in Science?," The Review of Economics and Statistics, MIT Press, vol. 98(2), pages 397-414, May.
    17. Stephan, Paula E., 2010. "The Economics of Science," Handbook of the Economics of Innovation, in: Bronwyn H. Hall & Nathan Rosenberg (ed.), Handbook of the Economics of Innovation, edition 1, volume 1, chapter 0, pages 217-273, Elsevier.
    18. Benjamin Elsner & Ingo E. Isphording, 2017. "A Big Fish in a Small Pond: Ability Rank and Human Capital Investment," Journal of Labor Economics, University of Chicago Press, vol. 35(3), pages 787-828.
    19. Pierre Azoulay & Joshua S. Graff Zivin & Jialan Wang, 2010. "Superstar Extinction," The Quarterly Journal of Economics, President and Fellows of Harvard College, vol. 125(2), pages 549-589.
    20. Ashraf, Nava & Bandiera, Oriana & Jack, B. Kelsey, 2014. "No margin, no mission? A field experiment on incentives for public service delivery," Journal of Public Economics, Elsevier, vol. 120(C), pages 1-17.
    21. Jacob, Brian A. & Lefgren, Lars, 2011. "The impact of research grant funding on scientific productivity," Journal of Public Economics, Elsevier, vol. 95(9), pages 1168-1177.
    22. Muralidharan, Karthik & Sundararaman, Venkatesh, 2011. "Teacher opinions on performance pay: Evidence from India," Economics of Education Review, Elsevier, vol. 30(3), pages 394-403, June.
    23. Waverly W. Ding & Sharon G. Levin & Paula E. Stephan & Anne E. Winkler, 2010. "The Impact of Information Technology on Academic Scientists' Productivity and Collaboration Patterns," Management Science, INFORMS, vol. 56(9), pages 1439-1461, September.
    24. Butler, Linda, 2003. "Explaining Australia's increased share of ISI publications--the effects of a funding formula based on publication counts," Research Policy, Elsevier, vol. 32(1), pages 143-155, January.
    25. Aliakbar Akbaritabar & Giangiacomo Bravo & Flaminio Squazzoni, 2021. "The impact of a national research assessment on the publications of sociologists in Italy," Science and Public Policy, Oxford University Press, vol. 48(5), pages 662-678.
    26. Karthik Muralidharan & Venkatesh Sundararaman, 2011. "Teacher Performance Pay: Experimental Evidence from India," Journal of Political Economy, University of Chicago Press, vol. 119(1), pages 39-77.
    27. Scott Stern, 2004. "Do Scientists Pay to Be Scientists?," Management Science, INFORMS, vol. 50(6), pages 835-853, June.
    28. Abhijit V. Banerjee & Esther Duflo & Rachel Glennerster, 2008. "Putting a Band-Aid on a Corpse: Incentives for Nurses in the Indian Public Health Care System," Journal of the European Economic Association, MIT Press, vol. 6(2-3), pages 487-500, 04-05.
    29. Richard Van Noorden, 2013. "Brazilian citation scheme outed," Nature, Nature, vol. 500(7464), pages 510-511, August.
    30. Duffy, John & Kornienko, Tatiana, 2010. "Does competition affect giving?," Journal of Economic Behavior & Organization, Elsevier, vol. 74(1-2), pages 82-103, May.
    31. repec:adr:anecst:y:2005:i:79-80:p:04 is not listed on IDEAS
    32. Hicks, Diana, 2012. "Performance-based university research funding systems," Research Policy, Elsevier, vol. 41(2), pages 251-261.
    33. Kyle Myers, 2020. "The Elasticity of Science," American Economic Journal: Applied Economics, American Economic Association, vol. 12(4), pages 103-134, October.
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Irawan, Dasapta Erwin & Abraham, Juneman & Ridlo, Ilham Akhsanu & Nugraha, Harya Dwi & Wekke, Ismail Suardi & Fuad, Anis & Rudyawan, Alfend & Seftyono, Cahyo, 2023. "Comments on “Ranking researchers: Evidence from Indonesia” by Fry et al. (2023)," Research Policy, Elsevier, vol. 52(7).

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Aparicio Fenoll, Ainoa, 2021. "The best in the class," Economics of Education Review, Elsevier, vol. 84(C).
    2. Katrin Hussinger & João N. Carvalho, 2022. "The long-term effect of research grants on the scientific output of university professors," Industry and Innovation, Taylor & Francis Journals, vol. 29(4), pages 463-487, April.
    3. Murphy, Richard & Weinhardt, Felix, 2020. "Top of the Class: The Importance of Ordinal Rank," EconStor Open Access Articles and Book Chapters, ZBW - Leibniz Information Centre for Economics, vol. 87(6), pages 2777-2826.
    4. Ayoubi, Charles & Pezzoni, Michele & Visentin, Fabiana, 2019. "The important thing is not to win, it is to take part: What if scientists benefit from participating in research grant competitions?," Research Policy, Elsevier, vol. 48(1), pages 84-97.
    5. Delavallade,Clara Anne, 2021. "Motivating Teams : Private Feedback and Public Recognition at Work," Policy Research Working Paper Series 9621, The World Bank.
    6. Fischer, Mira & Wagner, Valentin, 2018. "Effects of Timing and Reference Frame of Feedback: Evidence from a Field Experiment," IZA Discussion Papers 11970, Institute of Labor Economics (IZA).
    7. Delavallade, Clara, 2021. "Motivating teams: Private feedback and public recognition at work," Journal of Public Economics, Elsevier, vol. 197(C).
    8. Aisha J Ali & Javier Fuenzalida & Margarita Gómez & Martin J Williams, 2021. "Four lenses on people management in the public sector: an evidence review and synthesis," Oxford Review of Economic Policy, Oxford University Press and Oxford Review of Economic Policy Limited, vol. 37(2), pages 335-366.
    9. Yu, Han, 2020. "Am I the big fish? The effect of ordinal rank on student academic performance in middle school," Journal of Economic Behavior & Organization, Elsevier, vol. 176(C), pages 18-41.
    10. Rigissa Megalokonomou & Yi Zhang, 2022. "How Good Am I? Effects and Mechanisms behind Salient Ranks," CESifo Working Paper Series 9991, CESifo.
    11. Abramo, Giovanni & D'Angelo, Ciriaco Andrea & Grilli, Leonardo, 2021. "The effects of citation-based research evaluation schemes on self-citation behavior," Journal of Informetrics, Elsevier, vol. 15(4).
    12. Berlinski, Samuel & Ramos, Alejandra, 2020. "Teacher mobility and merit pay: Evidence from a voluntary public award program," Journal of Public Economics, Elsevier, vol. 186(C).
    13. Rigissa Megalokonomou & Yi Zhang, 2023. "How Good Am I? Effects and Mechanisms Behind Salient Rank," Monash Economics Working Papers 2023-07, Monash University, Department of Economics.
    14. Benjamin Elsner & Ingo E Isphording & Ulf Zölitz, 2021. "Achievement Rank Affects Performance and Major Choices in College [Gender, competitiveness, and socialization at a young age: evidence from a matrilineal and a patriarchal society]," The Economic Journal, Royal Economic Society, vol. 131(640), pages 3182-3206.
    15. Hottenrott, Hanna & Lawson, Cornelia, 2013. "Fishing for Complementarities: Competitive Research Funding and Research Productivity," Department of Economics and Statistics Cognetti de Martiis LEI & BRICK - Laboratory of Economics of Innovation "Franco Momigliano", Bureau of Research in Innovation, Complexity and Knowledge, Collegio 201318, University of Turin.
    16. Arora, Puneet & Musaddiq, Tareena, 2023. "Can rank-based non-monetary rewards improve student attendance? Experimental evidence from India," Journal of Behavioral and Experimental Economics (formerly The Journal of Socio-Economics), Elsevier, vol. 107(C).
    17. David Gill & Zdenka Kissová & Jaesun Lee & Victoria Prowse, 2019. "First-Place Loving and Last-Place Loathing: How Rank in the Distribution of Performance Affects Effort Provision," Management Science, INFORMS, vol. 65(2), pages 494-507, February.
    18. Anckaert, Paul-Emmanuel & Cassiman, David & Cassiman, Bruno, 2020. "Fostering practice-oriented and use-inspired science in biomedical research," Research Policy, Elsevier, vol. 49(2).
    19. Sander Gerritsen & Karen van der Wiel & Erik Plug, 2013. "Up or out? How individual research grants affect academic careers in the Netherlands," CPB Discussion Paper 249.rdf, CPB Netherlands Bureau for Economic Policy Analysis.
    20. Seokbeom Kwon & Kazuyuki Motohashi & Kenta Ikeuchi, 2022. "Chasing two hares at once? Effect of joint institutional change for promoting commercial use of university knowledge and scientific research," The Journal of Technology Transfer, Springer, vol. 47(4), pages 1242-1272, August.

    More about this item

    Keywords

    Innovation; Development; Technological change; Scientific research;
    All these keywords.

    JEL classification:

    • O1 - Economic Development, Innovation, Technological Change, and Growth - - Economic Development
    • O3 - Economic Development, Innovation, Technological Change, and Growth - - Innovation; Research and Development; Technological Change; Intellectual Property Rights
    • O33 - Economic Development, Innovation, Technological Change, and Growth - - Innovation; Research and Development; Technological Change; Intellectual Property Rights - - - Technological Change: Choices and Consequences; Diffusion Processes
    • O38 - Economic Development, Innovation, Technological Change, and Growth - - Innovation; Research and Development; Technological Change; Intellectual Property Rights - - - Government Policy
    • I23 - Health, Education, and Welfare - - Education - - - Higher Education; Research Institutions

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:eee:respol:v:52:y:2023:i:5:s0048733323000379. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Catherine Liu (email available below). General contact details of provider: http://www.elsevier.com/locate/respol .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.