IDEAS home Printed from
   My bibliography  Save this article

A numerical algorithm with preference statements to evaluate the performance of scientists


  • Martin Ricker

    (Universidad Nacional Autónoma de México (UNAM))


Academic evaluation committees have been increasingly receptive for using the number of published indexed articles, as well as citations, to evaluate the performance of scientists. It is, however, impossible to develop a stand-alone, objective numerical algorithm for the evaluation of academic activities, because any evaluation necessarily includes subjective preference statements. In a market, the market prices represent preference statements, but scientists work largely in a non-market context. I propose a numerical algorithm that serves to determine the distribution of reward money in Mexico’s evaluation system, which uses relative prices of scientific goods and services as input. The relative prices would be determined by an evaluation committee. In this way, large evaluation systems (like Mexico’s Sistema Nacional de Investigadores) could work semi-automatically, but not arbitrarily or superficially, to determine quantitatively the academic performance of scientists every few years. Data of 73 scientists from the Biology Institute of Mexico’s National University are analyzed, and it is shown that the reward assignation and academic priorities depend heavily on those preferences. A maximum number of products or activities to be evaluated is recommended, to encourage quality over quantity.

Suggested Citation

  • Martin Ricker, 2015. "A numerical algorithm with preference statements to evaluate the performance of scientists," Scientometrics, Springer;Akadémiai Kiadó, vol. 103(1), pages 191-212, April.
  • Handle: RePEc:spr:scient:v:103:y:2015:i:1:d:10.1007_s11192-014-1521-2
    DOI: 10.1007/s11192-014-1521-2

    Download full text from publisher

    File URL:
    File Function: Abstract
    Download Restriction: Access to the full text of the articles in this series is restricted.

    File URL:
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item

    As the access to this document is restricted, you may want to search for a different version of it.

    References listed on IDEAS

    1. Gonzalez-Brambila, Claudia & Veloso, Francisco M., 2007. "The determinants of research output and impact: A study of Mexican researchers," Research Policy, Elsevier, vol. 36(7), pages 1035-1051, September.
    2. David Roessner, 2000. "Quantitative and qualitative methods and measures in the evaluation of research," Research Evaluation, Oxford University Press, vol. 9(2), pages 125-132, August.
    3. Waltman, Ludo & van Eck, Nees Jan & Wouters, Paul, 2013. "Counting publications and citations: Is more always better?," Journal of Informetrics, Elsevier, vol. 7(3), pages 635-641.
    4. Korhonen, Pekka & Tainio, Risto & Wallenius, Jyrki, 2001. "Value efficiency analysis of academic research," European Journal of Operational Research, Elsevier, vol. 130(1), pages 121-132, April.
    5. Stevie Upton & Paul Vallance & John Goddard, 2014. "From outcomes to process: evidence for a new approach to research impact assessment," Research Evaluation, Oxford University Press, vol. 23(4), pages 352-365.
    6. Gagolewski, Marek, 2013. "Scientific impact assessment cannot be fair," Journal of Informetrics, Elsevier, vol. 7(4), pages 792-802.
    7. Ricker, Martin, 1997. "Limits to economic growth as shown by a computable general equilibrium model," Ecological Economics, Elsevier, vol. 21(2), pages 141-158, May.
    Full references (including those not matched with items on IDEAS)

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Martin Ricker, 2017. "Letter to the Editor: About the quality and impact of scientific articles," Scientometrics, Springer;Akadémiai Kiadó, vol. 111(3), pages 1851-1855, June.
    2. Goodall, Amanda H., 2009. "Highly cited leaders and the performance of research universities," Research Policy, Elsevier, vol. 38(7), pages 1079-1092, September.
    3. David Grosse Kathoefer & Jens Leker, 2012. "Knowledge transfer in academia: an exploratory study on the Not-Invented-Here Syndrome," The Journal of Technology Transfer, Springer, vol. 37(5), pages 658-675, October.
    4. Vanessa Sandoval-Romero & Vincent Larivière, 2020. "The national system of researchers in Mexico: implications of publication incentives for researchers in social sciences," Scientometrics, Springer;Akadémiai Kiadó, vol. 122(1), pages 99-126, January.
    5. Rafols, Ismael & Leydesdorff, Loet & O’Hare, Alice & Nightingale, Paul & Stirling, Andy, 2012. "How journal rankings can suppress interdisciplinary research: A comparison between Innovation Studies and Business & Management," Research Policy, Elsevier, vol. 41(7), pages 1262-1282.
    6. Aboal, Diego & Tacsir, Ezequiel, 2016. "The impact of ex-ante subsidies to researchers on researcher's productivity: Evidence from a developing country," MERIT Working Papers 2016-019, United Nations University - Maastricht Economic and Social Research Institute on Innovation and Technology (MERIT).
    7. van den Besselaar, Peter & Sandström, Ulf, 2015. "Early career grants, performance, and careers: A study on predictive validity of grant decisions," Journal of Informetrics, Elsevier, vol. 9(4), pages 826-838.
    8. Lorena Rivera León & Jacques Mairesse & Robin Cowan, 2017. "Gender Gaps and Scientific Productivity in Middle-Income Countries: Evidence from Mexico," IDB Publications (Working Papers) 98456, Inter-American Development Bank.
    9. Mehdi Rhaiem & Nabil Amara, 2020. "Determinants of research efficiency in Canadian business schools: evidence from scholar-level data," Scientometrics, Springer;Akadémiai Kiadó, vol. 125(1), pages 53-99, October.
    10. Johnes, Jill & Yu, Li, 2008. "Measuring the research performance of Chinese higher education institutions using data envelopment analysis," China Economic Review, Elsevier, vol. 19(4), pages 679-696, December.
    11. Gowdy, John M. & Ferreri Carbonell, Ada, 1999. "Toward consilience between biology and economics: the contribution of Ecological Economics," Ecological Economics, Elsevier, vol. 29(3), pages 337-348, June.
    12. Fu, Junying & Frietsch, Rainer & Tagscherer, Ulrike, 2013. "Publication activity in the Science Citation Index Expanded (SCIE) database in the context of Chinese science and technology policy from 1977 to 2012," Discussion Papers "Innovation Systems and Policy Analysis" 35, Fraunhofer Institute for Systems and Innovation Research (ISI).
    13. Rivera-Huerta, René & Dutrénit, Gabriela & Ekboir, Javier Mario & Sampedro, José Luis & Vera-Cruz, Alexandre O., 2011. "Do linkages between farmers and academic researchers influence researcher productivity? The Mexican case," Research Policy, Elsevier, vol. 40(7), pages 932-942, September.
    14. Tavana, Madjid & Ebrahimnejad, Ali & Santos-Arteaga, Francisco J. & Mansourzadeh, Seyed Mehdi & Matin, Reza Kazemi, 2018. "A hybrid DEA-MOLP model for public school assessment and closure decision in the City of Philadelphia," Socio-Economic Planning Sciences, Elsevier, vol. 61(C), pages 70-89.
    15. Young-Don Cho & Hoo-Gon Choi, 2013. "Principal parameters affecting R&D exploitation of nanotechnology research: a case for Korea," Scientometrics, Springer;Akadémiai Kiadó, vol. 96(3), pages 881-899, September.
    16. Claudia N Gonzalez-Brambila & Leonardo Reyes-Gonzalez & Francisco Veloso & Miguel Angel Perez-Angón, 2016. "The Scientific Impact of Developing Nations," PLOS ONE, Public Library of Science, vol. 11(3), pages 1-14, March.
    17. George E. Halkos & Nickolaos G. Tzeremes & Stavros A. Kourtzidis, 2010. "An application of statistical interference in DEA models: An analysis of public owned university departments' efficiency," EERI Research Paper Series EERI_RP_2010_17, Economics and Econometrics Research Institute (EERI), Brussels.
    18. Zhang, Daqun & Banker, Rajiv D. & Li, Xiaoxuan & Liu, Wenbin, 2011. "Performance impact of research policy at the Chinese Academy of Sciences," Research Policy, Elsevier, vol. 40(6), pages 875-885, July.
    19. Tol, Richard S.J., 2013. "The Matthew effect for cohorts of economists," Journal of Informetrics, Elsevier, vol. 7(2), pages 522-527.
    20. João M. Santos & Hugo Horta, 2015. "The generational gap of science: a dynamic cluster analysis of doctorates in an evolving scientific system," Scientometrics, Springer;Akadémiai Kiadó, vol. 104(1), pages 381-406, July.


    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:spr:scient:v:103:y:2015:i:1:d:10.1007_s11192-014-1521-2. See general information about how to correct material in RePEc.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: . General contact details of provider: .

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Sonal Shukla or Springer Nature Abstracting and Indexing (email available below). General contact details of provider: .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service hosted by the Research Division of the Federal Reserve Bank of St. Louis . RePEc uses bibliographic data supplied by the respective publishers.