IDEAS home Printed from https://ideas.repec.org/a/spr/scient/v88y2011i3d10.1007_s11192-011-0440-8.html
   My bibliography  Save this article

Scientometric impact assessment of a research policy instrument: the case of rating researchers on scientific outputs in South Africa

Author

Listed:
  • Roula Inglesi-Lotz

    (University of Pretoria)

  • Anastassios Pouris

    (University of Pretoria)

Abstract

The influence of the National Research Foundation’s (NRF) rating system on the productivity of the South African social science researchers is investigated scientometrically for the period from 1981 to 2006. Their output performance is mainly indicated by their research publications. Following international best practice in scientometrics as well as the behavioural reinforcement theory, we employed the “before/after control impact (BACI) method”, as well as the well known econometric breakpoint test as proposed by Chow. We use as control group the publications in the field of clinical medicine. The field is not supported by NRF and hence clinical medicine researchers are not affected by the evaluation and rating system. The findings show a positive impact of the NRF programme on the research outputs of social sciences researchers and the implementation of the programme has increased the relevant population of research articles by an average of 24.5% (during the first 5 years) over the expected number of publication without the programme. The results confirm the scientometric findings of other studies (e.g. that of Nederhof) that ratings promulgate research productivity.

Suggested Citation

  • Roula Inglesi-Lotz & Anastassios Pouris, 2011. "Scientometric impact assessment of a research policy instrument: the case of rating researchers on scientific outputs in South Africa," Scientometrics, Springer;Akadémiai Kiadó, vol. 88(3), pages 747-760, September.
    • Handle: RePEc:spr:scient:v:88:y:2011:i:3:d:10.1007_s11192-011-0440-8
    DOI: 10.1007/s11192-011-0440-8
    as

    Download full text from publisher

    File URL: http://link.springer.com/10.1007/s11192-011-0440-8
    File Function: Abstract
    Download Restriction: Access to the full text of the articles in this series is restricted.
    File URL: https://libkey.io/10.1007/s11192-011-0440-8?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    As the access to this document is restricted, you may want to search for a different version of it.

    References listed on IDEAS

    as
    1. Marianne Bertrand & Simeon Djankov & Rema Hanna & Sendhil Mullainathan, 2006. "Does corruption produce unsafe drivers?," Natural Field Experiments 00218, The Field Experiments Website.
    2. Sandra Miguel & Félix Moya-Anegón & Víctor Herrero-Solana, 2010. "The impact of the socio-economic crisis of 2001 on the scientific system of Argentina from the scientometric perspective," Scientometrics, Springer;Akadémiai Kiadó, vol. 85(2), pages 495-507, November.
    3. Duflo, Esther & Glennerster, Rachel & Kremer, Michael, 2008. "Using Randomization in Development Economics Research: A Toolkit," Handbook of Development Economics, in: T. Paul Schultz & John A. Strauss (ed.), Handbook of Development Economics, edition 1, volume 4, chapter 61, pages 3895-3962, Elsevier.
    4. Andrew Leigh, 2003. "Employment Effects of Minimum Wages: Evidence from a Quasi‐Experiment," Australian Economic Review, The University of Melbourne, Melbourne Institute of Applied Economic and Social Research, vol. 36(4), pages 361-373, December.
    5. Anton J. Nederhof, 2008. "Policy impact of bibliometric rankings of research performance of departments and individuals in economics," Scientometrics, Springer;Akadémiai Kiadó, vol. 74(1), pages 163-174, January.
    6. Mee-Jean Kim, 2007. "A bibliometric analysis of the effectiveness of Korea’s Biotechnology Stimulation Plans, with a comparison with four other Asian nations," Scientometrics, Springer;Akadémiai Kiadó, vol. 72(3), pages 371-388, September.
    7. David A Raitzer & Timothy G Kelley, 2008. "Assessing the contribution of impact assessment to donor decisions for international agricultural research," Research Evaluation, Oxford University Press, vol. 17(3), pages 187-199, September.
    8. Michael Kremer, 2003. "Randomized Evaluations of Educational Programs in Developing Countries: Some Lessons," American Economic Review, American Economic Association, vol. 93(2), pages 102-106, May.
    9. Groot, Tom & Garcia-Valderrama, Teresa, 2006. "Research quality and efficiency: An analysis of assessments and management issues in Dutch economics and business research programs," Research Policy, Elsevier, vol. 35(9), pages 1362-1376, November.
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Nelson Casimiro Zavale & Patrício Vitorino Langa, 2018. "University-industry linkages’ literature on Sub-Saharan Africa: systematic literature review and bibliometric account," Scientometrics, Springer;Akadémiai Kiadó, vol. 116(1), pages 1-49, July.
    2. Anastassios Pouris, 2012. "Scientometric research in South Africa and successful policy instruments," Scientometrics, Springer;Akadémiai Kiadó, vol. 91(2), pages 317-325, May.
    3. Matteo Pedrini & Valentina Langella & Mario Alberto Battaglia & Paola Zaratin, 2018. "Assessing the health research’s social impact: a systematic review," Scientometrics, Springer;Akadémiai Kiadó, vol. 114(3), pages 1227-1250, March.
    4. Roula Inglesi-Lotz & Mehmet Balcilar & Rangan Gupta, 2014. "Time-varying causality between research output and economic growth in US," Scientometrics, Springer;Akadémiai Kiadó, vol. 100(1), pages 203-216, July.
    5. R. Inglesi-Lotz & A. Pouris, 2013. "The influence of scientific research output of academics on economic growth in South Africa: an autoregressive distributed lag (ARDL) application," Scientometrics, Springer;Akadémiai Kiadó, vol. 95(1), pages 129-139, April.
    6. Similo Ngwenya & Nelius Boshoff, 2022. "Different manifestations of ‘context’: examples from a bibliometric study of research in Zimbabwe in Southern Africa," Scientometrics, Springer;Akadémiai Kiadó, vol. 127(7), pages 3911-3933, July.
    7. Fernando A. B. Colugnati & Sergio Firpo & Paula F. Drummond Castro & Juan E. Sepulveda & Sergio L. M. Salles-Filho, 2014. "A propensity score approach in the impact evaluation on scientific production in Brazilian biodiversity research: the BIOTA Program," Scientometrics, Springer;Akadémiai Kiadó, vol. 101(1), pages 85-107, October.
    8. Fiorentin, Florencia & Pereira, Mariano & Suárez, Diana, 2022. "The Gender Gap in Public S&T Funding: The Matilda Effect in STEM Disciplines in Argentina," IDB Publications (Working Papers) 11990, Inter-American Development Bank.

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Duflo, Esther & Glennerster, Rachel & Kremer, Michael, 2008. "Using Randomization in Development Economics Research: A Toolkit," Handbook of Development Economics, in: T. Paul Schultz & John A. Strauss (ed.), Handbook of Development Economics, edition 1, volume 4, chapter 61, pages 3895-3962, Elsevier.
    2. Eduard Marinov, 2019. "The 2019 Nobel Prize in Economics," Economic Thought journal, Bulgarian Academy of Sciences - Economic Research Institute, issue 6, pages 78-116.
    3. Sawada Yasuyuki & Mahmud Minhaj & Seki Mai & Le An & Kawarazaki Hikaru, 2017. "Individualized Self-learning Program to Improve Primary Education: Evidence from a Randomized Field Experiment in Bangladesh," Working Papers 156, JICA Research Institute.
    4. Nikoloz Kudashvili, 2018. "Sources of Statistical Discrimination: Experimental Evidence from Georgia," CERGE-EI Working Papers wp612, The Center for Economic Research and Graduate Education - Economics Institute, Prague.
    5. Jesse Cunha, 2010. "Testing Paternalism: Cash vs. In-kind Transfer in Rural Mexico," Discussion Papers 09-021, Stanford Institute for Economic Policy Research.
    6. Miriam Bruhn & David McKenzie, 2009. "In Pursuit of Balance: Randomization in Practice in Development Field Experiments," American Economic Journal: Applied Economics, American Economic Association, vol. 1(4), pages 200-232, October.
    7. Committee, Nobel Prize, 2019. "Understanding development and poverty alleviation," Nobel Prize in Economics documents 2019-2, Nobel Prize Committee.
    8. Awotide, Bola Amoke & Awoyemi, Taiwo Timothy & Diagne, Aliou & Ojehomon, Vivian E.T., 2012. "Impact of Seed Voucher System on Rice Farmers’ Welfare in Nigeria: A Randomized Control Trial Approach," 2012 Conference, August 18-24, 2012, Foz do Iguacu, Brazil 124995, International Association of Agricultural Economists.
    9. Clarke, Damian, 2019. "A convenient omitted variable bias formula for treatment effect models," Economics Letters, Elsevier, vol. 174(C), pages 84-88.
    10. Sawada, Yasuyuki & Ishii, Takaharu, 2012. "Do Community-Managed Schools Facilitate Social Capital Accumulation? Evidence from the COGES Project in Burkina Faso," Working Papers 42, JICA Research Institute.
    11. Emmanuel Jimenez & Yasuyuki Sawada, 2014. "Does Community Management Help Keep Children in Schools? Evidence Using Panel Data from El Salvador's EDUCO Program," Economic Development and Cultural Change, University of Chicago Press, vol. 62(2), pages 307-338.
    12. David K. Evans & Arkadipta Ghosh, 2008. "Prioritizing Educational Investments in Children in the Developing World," Working Papers WR-587, RAND Corporation.
    13. Eduardo Fernández-Arias & Ricardo Hausmann & Ugo Panizza, 2020. "Smart Development Banks," Journal of Industry, Competition and Trade, Springer, vol. 20(2), pages 395-420, June.
    14. Ashish Arora & Michelle Gittelman & Sarah Kaplan & John Lynch & Will Mitchell & Nicolaj Siggelkow & Aaron K. Chatterji & Michael Findley & Nathan M. Jensen & Stephan Meier & Daniel Nielson, 2016. "Field experiments in strategy research," Strategic Management Journal, Wiley Blackwell, vol. 37(1), pages 116-132, January.
    15. Patrick Guillaumont, 2011. "Aid effectiveness for poverty reduction:macroeconomic overview and emerging issues," CERDI Working papers halshs-00554285, HAL.
    16. Martin Schlotter & Guido Schwerdt & Ludger Woessmann, 2011. "Econometric methods for causal evaluation of education policies and practices: a non-technical guide," Education Economics, Taylor & Francis Journals, vol. 19(2), pages 109-137.
    17. Olken, Benjamin A., 2009. "Corruption perceptions vs. corruption reality," Journal of Public Economics, Elsevier, vol. 93(7-8), pages 950-964, August.
    18. María laura Alzúa & Guillermo Cruces & Carolina Lopez, 2016. "Long-Run Effects Of Youth Training Programs: Experimental Evidence From Argentina," Economic Inquiry, Western Economic Association International, vol. 54(4), pages 1839-1859, October.
    19. Antonia Grohmann & Lukas Menkhoff & Helke Seitz, 2022. "The Effect of Personalized Feedback on Small Enterprises’ Finances in Uganda," Economic Development and Cultural Change, University of Chicago Press, vol. 70(3), pages 1197-1227.
    20. Pedro Carneiro & Sokbae Lee & Daniel Wilhelm, 2020. "Optimal data collection for randomized control trials [Microcredit impacts: Evidence from a randomized microcredit program placement experiment by Compartamos Banco]," The Econometrics Journal, Royal Economic Society, vol. 23(1), pages 1-31.

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:spr:scient:v:88:y:2011:i:3:d:10.1007_s11192-011-0440-8. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Sonal Shukla or Springer Nature Abstracting and Indexing (email available below). General contact details of provider: http://www.springer.com .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.