The predictive validity of peer review: A selective review of the judgmental forecasting qualities of peers, and implications for innovation in science
Author
Abstract
Suggested Citation
DOI: 10.1016/j.ijforecast.2010.03.003
Download full text from publisher
As the access to this document is restricted, you may want to search for a different version of it.
References listed on IDEAS
- Steven D. Levitt & John A. List, 2011.
"Was There Really a Hawthorne Effect at the Hawthorne Plant? An Analysis of the Original Illumination Experiments,"
American Economic Journal: Applied Economics, American Economic Association, vol. 3(1), pages 224-238, January.
- Steven D. Levitt & John A. List, 2009. "Was there Really a Hawthorne Effect at the Hawthorne Plant? An Analysis of the Original Illumination Experiments," NBER Working Papers 15016, National Bureau of Economic Research, Inc.
- Bornmann, Lutz & Daniel, Hans-Dieter, 2009. "Extent of type I and type II errors in editorial decisions: A case study on Angewandte Chemie International Edition," Journal of Informetrics, Elsevier, vol. 3(4), pages 348-352.
- Michael Obrecht & Karl Tibelius & Guy D'Aloisio, 2007. "Examining the value added by committee discussion in the review of applications for research awards," Research Evaluation, Oxford University Press, vol. 16(2), pages 79-91, June.
- J. Scott Armstrong & Ruth Pagell, 2003. "The Ombudsman: Reaping Benefits from Management Research: Lessons from the Forecasting Principles Project," Interfaces, INFORMS, vol. 33(6), pages 91-111, December.
- Herbert Marsh & Lutz Bornmann, 2009. "Do women have less success in peer review?," Nature, Nature, vol. 459(7246), pages 602-602, May.
- Lutz Bornmann & Gerlind Wallon & Anna Ledin, 2008. "Does the Committee Peer Review Select the Best Applicants for Funding? An Investigation of the Selection Process for Two European Molecular Biology Organization Programmes," PLOS ONE, Public Library of Science, vol. 3(10), pages 1-11, October.
- Sniezek, Janet A. & Buckley, Timothy, 1995. "Cueing and Cognitive Conflict in Judge-Advisor Decision Making," Organizational Behavior and Human Decision Processes, Elsevier, vol. 62(2), pages 159-174, May.
- Amos Tversky & Daniel Kahneman, 1991. "Loss Aversion in Riskless Choice: A Reference-Dependent Model," The Quarterly Journal of Economics, President and Fellows of Harvard College, vol. 106(4), pages 1039-1061.
- repec:bla:jamist:v:58:y:2007:i:5:p:734-743 is not listed on IDEAS
- Lawrence, Michael & Goodwin, Paul & O'Connor, Marcus & Onkal, Dilek, 2006. "Judgmental forecasting: A review of progress over the last 25 years," International Journal of Forecasting, Elsevier, vol. 22(3), pages 493-518.
- Henk F Moed, 2007. "The future of research evaluation rests with an intelligent combination of advanced metrics and transparent peer review," Science and Public Policy, Oxford University Press, vol. 34(8), pages 575-583, October.
- Liv Langfeldt, 2006. "The policy challenges of peer review: managing bias, conflict of interests and interdisciplinary assessments," Research Evaluation, Oxford University Press, vol. 15(1), pages 31-41, April.
- Sune Lehmann & Andrew D. Jackson & Benny E. Lautrup, 2006. "Measures for measures," Nature, Nature, vol. 444(7122), pages 1003-1004, December.
- Lutz Bornmann & Hans-Dieter Daniel, 2006. "Selecting scientific excellence through committee peer review - A citation analysis of publications previously published to approval or rejection of post-doctoral research fellowship applicants," Scientometrics, Springer;Akadémiai Kiadó, vol. 68(3), pages 427-440, September.
- Joshua S. Gans & George B. Shepherd, 1994. "How Are the Mighty Fallen: Rejected Classic Articles by Leading Economists," Journal of Economic Perspectives, American Economic Association, vol. 8(1), pages 165-179, Winter.
- repec:bla:jamest:v:45:y:1994:i:1:p:1-11 is not listed on IDEAS
- Göran Melin & Rickard Danell, 2006. "The top eight percent: Development of approved and rejected applicants for a prestigious grant in Sweden," Science and Public Policy, Oxford University Press, vol. 33(10), pages 702-712, December.
- Peter A. Lawrence, 2003. "The politics of publication," Nature, Nature, vol. 422(6929), pages 259-261, March.
- David N. Laband, 1990. "Is There Value-Added from the Review Process in Economics?: Preliminary Evidence from Authors," The Quarterly Journal of Economics, President and Fellows of Harvard College, vol. 105(2), pages 341-352.
- Samuelson, William & Zeckhauser, Richard, 1988. "Status Quo Bias in Decision Making," Journal of Risk and Uncertainty, Springer, vol. 1(1), pages 7-59, March.
- Peter van den Besselaar & Loet Leydesdorff, 2009. "Past performance, peer review and project selection: a case study in the social and behavioral sciences," Research Evaluation, Oxford University Press, vol. 18(4), pages 273-288, October.
- Anthony F. J. van Raan, 2004. "Sleeping Beauties in science," Scientometrics, Springer;Akadémiai Kiadó, vol. 59(3), pages 467-472, March.
- Liv Langfeldt, 2004. "Expert panels evaluating research: decision-making and sources of bias," Research Evaluation, Oxford University Press, vol. 13(1), pages 51-62, April.
- William H. Starbuck, 2005. "How Much Better Are the Most-Prestigious Journals? The Statistics of Academic Publication," Organization Science, INFORMS, vol. 16(2), pages 180-200, April.
- Wright, George & Lawrence, Michael J. & Collopy, Fred, 1996. "The role and validity of judgment in forecasting," International Journal of Forecasting, Elsevier, vol. 12(1), pages 1-8, March.
- Lutz Bornmann & Hans-Dieter Daniel, 2005. "Selection of research fellowship recipients by committee peer review. Reliability, fairness and predictive validity of Board of Trustees' decisions," Scientometrics, Springer;Akadémiai Kiadó, vol. 63(2), pages 297-320, April.
- Upali W. Jayasinghe & Herbert W. Marsh & Nigel Bond, 2006. "A new reader trial approach to peer review in funding research grants: An Australian experiment," Scientometrics, Springer;Akadémiai Kiadó, vol. 69(3), pages 591-606, December.
- Bonaccio, Silvia & Dalal, Reeshad S., 2006. "Advice taking and decision-making: An integrative literature review, and implications for the organizational sciences," Organizational Behavior and Human Decision Processes, Elsevier, vol. 101(2), pages 127-151, November.
- repec:bla:jamist:v:59:y:2008:i:11:p:1841-1852 is not listed on IDEAS
- J. Scott Armstrong, 2006. "Should the Forecasting Process Eliminate Face-to-Face Meetings?," Foresight: The International Journal of Applied Forecasting, International Institute of Forecasters, issue 5, pages 3-8, Fall.
Citations
Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
Cited by:
- van den Besselaar, Peter & Sandström, Ulf, 2015. "Early career grants, performance, and careers: A study on predictive validity of grant decisions," Journal of Informetrics, Elsevier, vol. 9(4), pages 826-838.
- Wright, George & Rowe, Gene, 2011. "Group-based judgmental forecasting: An integration of extant knowledge and the development of priorities for a new research agenda," International Journal of Forecasting, Elsevier, vol. 27(1), pages 1-13, January.
- Lutz Bornmann, 2015. "Interrater reliability and convergent validity of F1000Prime peer review," Journal of the Association for Information Science & Technology, Association for Information Science & Technology, vol. 66(12), pages 2415-2426, December.
- Girish Mallapragada & Nandini Lahiri & Atul Nerkar, 2016. "Peer Review and Research Impact," Customer Needs and Solutions, Springer;Institute for Sustainable Innovation and Growth (iSIG), vol. 3(1), pages 29-41, March.
- Gill, Chelsea & Mehrotra, Vishal & Moses, Olayinka & Bui, Binh, 2023. "The impact of the pitching research framework on AFAANZ grant applications," Pacific-Basin Finance Journal, Elsevier, vol. 77(C).
- repec:eee:intfor:v:27:y:2011:i:1:p:1-13 is not listed on IDEAS
- Girish Mallapragada & Nandini Lahiri & Atul Nerkar, 2016. "Peer Review and Research Impact," Customer Needs and Solutions, Springer;Institute for Sustainable Innovation and Growth (iSIG), vol. 3(1), pages 29-41, March.
- Jens Jirschitzka & Aileen Oeberst & Richard Göllner & Ulrike Cress, 2017. "Inter-rater reliability and validity of peer reviews in an interdisciplinary field," Scientometrics, Springer;Akadémiai Kiadó, vol. 113(2), pages 1059-1092, November.
- Richard R Snell, 2015. "Menage a Quoi? Optimal Number of Peer Reviewers," PLOS ONE, Public Library of Science, vol. 10(4), pages 1-14, April.
- Frederik T. Verleysen & Tim C.E. Engels, 2013. "A label for peer-reviewed books," Journal of the Association for Information Science & Technology, Association for Information Science & Technology, vol. 64(2), pages 428-430, February.
- Lutz Bornmann, 2012. "The Hawthorne effect in journal peer review," Scientometrics, Springer;Akadémiai Kiadó, vol. 91(3), pages 857-862, June.
- Krist Vaesen & Joel Katzav, 2017. "How much would each researcher receive if competitive government research funding were distributed equally among researchers?," PLOS ONE, Public Library of Science, vol. 12(9), pages 1-11, September.
Most related items
These are the items that most often cite the same works as this one and are cited by the same works as this one.- Benda, Wim G.G. & Engels, Tim C.E., 2011. "The predictive validity of peer review: A selective review of the judgmental forecasting qualities of peers, and implications for innovation in science," International Journal of Forecasting, Elsevier, vol. 27(1), pages 166-182, January.
- Kevin W. Boyack & Caleb Smith & Richard Klavans, 2018. "Toward predicting research proposal success," Scientometrics, Springer;Akadémiai Kiadó, vol. 114(2), pages 449-461, February.
- Vieira, Elizabeth S. & Cabral, José A.S. & Gomes, José A.N.F., 2014. "How good is a model based on bibliometric indicators in predicting the final decisions made by peers?," Journal of Informetrics, Elsevier, vol. 8(2), pages 390-405.
- van den Besselaar, Peter, 2012. "Selection committee membership: Service or self-service," Journal of Informetrics, Elsevier, vol. 6(4), pages 580-585.
- van den Besselaar, Peter & Sandström, Ulf, 2015. "Early career grants, performance, and careers: A study on predictive validity of grant decisions," Journal of Informetrics, Elsevier, vol. 9(4), pages 826-838.
- Van Swol, Lyn M., 2011. "Forecasting another’s enjoyment versus giving the right answer: Trust, shared values, task effects, and confidence in improving the acceptance of advice," International Journal of Forecasting, Elsevier, vol. 27(1), pages 103-120.
- Bar-Ilan, Judit, 2008. "Informetrics at the beginning of the 21st century—A review," Journal of Informetrics, Elsevier, vol. 2(1), pages 1-52.
- Mingers, John & Leydesdorff, Loet, 2015. "A review of theory and practice in scientometrics," European Journal of Operational Research, Elsevier, vol. 246(1), pages 1-19.
- Alvarado-Valencia, Jorge & Barrero, Lope H. & Önkal, Dilek & Dennerlein, Jack T., 2017. "Expertise, credibility of system forecasts and integration methods in judgmental demand forecasting," International Journal of Forecasting, Elsevier, vol. 33(1), pages 298-313.
- Margit Osterloh & Bruno S. Frey, 2009.
"Research Governance in Academia: Are there Alternatives to Academic Rankings?,"
CREMA Working Paper Series
2009-17, Center for Research in Economics, Management and the Arts (CREMA).
- Margit Osterloh & Bruno S. Frey, 2009. "Research governance in academia: are there alternatives to academic rankings?," IEW - Working Papers 423, Institute for Empirical Research in Economics - University of Zurich.
- Margit Osterloh & Bruno S. Frey, 2009. "Research Governance in Academia: Are there Alternatives to Academic Rankings?," CESifo Working Paper Series 2797, CESifo.
- Marsh, Herbert W. & Jayasinghe, Upali W. & Bond, Nigel W., 2011. "Gender differences in peer reviews of grant applications: A substantive-methodological synergy in support of the null hypothesis model," Journal of Informetrics, Elsevier, vol. 5(1), pages 167-180.
- Bornmann, Lutz & Leydesdorff, Loet & Van den Besselaar, Peter, 2010. "A meta-evaluation of scientific research proposals: Different ways of comparing rejected to awarded applications," Journal of Informetrics, Elsevier, vol. 4(3), pages 211-220.
- Filippo Radicchi & Claudio Castellano, 2013. "Analysis of bibliometric indicators for individual scholars in a large data set," Scientometrics, Springer;Akadémiai Kiadó, vol. 97(3), pages 627-637, December.
- Stefan Hornbostel & Susan Böhmer & Bernd Klingsporn & Jörg Neufeld & Markus Ins, 2009. "Funding of young scientist and scientific excellence," Scientometrics, Springer;Akadémiai Kiadó, vol. 79(1), pages 171-190, April.
- Van Swol, Lyn M., 2011. "Forecasting another's enjoyment versus giving the right answer: Trust, shared values, task effects, and confidence in improving the acceptance of advice," International Journal of Forecasting, Elsevier, vol. 27(1), pages 103-120, January.
- Petropoulos, Fotios & Apiletti, Daniele & Assimakopoulos, Vassilios & Babai, Mohamed Zied & Barrow, Devon K. & Ben Taieb, Souhaib & Bergmeir, Christoph & Bessa, Ricardo J. & Bijak, Jakub & Boylan, Joh, 2022.
"Forecasting: theory and practice,"
International Journal of Forecasting, Elsevier, vol. 38(3), pages 705-871.
- Fotios Petropoulos & Daniele Apiletti & Vassilios Assimakopoulos & Mohamed Zied Babai & Devon K. Barrow & Souhaib Ben Taieb & Christoph Bergmeir & Ricardo J. Bessa & Jakub Bijak & John E. Boylan & Jet, 2020. "Forecasting: theory and practice," Papers 2012.03854, arXiv.org, revised Jan 2022.
- Richard R Snell, 2015. "Menage a Quoi? Optimal Number of Peer Reviewers," PLOS ONE, Public Library of Science, vol. 10(4), pages 1-14, April.
- Steven M. Shugan, 2007. "The Editor's Secrets," Marketing Science, INFORMS, vol. 26(5), pages 589-595, 09-10.
- Osterloh, Margit & Frey, Bruno S., 2020. "How to avoid borrowed plumes in academia," Research Policy, Elsevier, vol. 49(1).
- Wiebke Roß & Jens Weghake, 2018. "Wa(h)re Liebe: Was Online-Dating-Plattformen über zweiseitige Märkte lehren," TUC Working Papers in Economics 0017, Abteilung für Volkswirtschaftslehre, Technische Universität Clausthal (Department of Economics, Technical University Clausthal).
More about this item
Keywords
Advice taking; Cognitive bias; Decision-making; Expert advice; Group decision making; Reliability;All these keywords.
Statistics
Access and download statisticsCorrections
All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:eee:intfor:v:27:y:2011:i:1:p:166-182. See general information about how to correct material in RePEc.
If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.
If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .
If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.
For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Catherine Liu (email available below). General contact details of provider: http://www.elsevier.com/locate/ijforecast .
Please note that corrections may take a couple of weeks to filter through the various RePEc services.