IDEAS home Printed from
MyIDEAS: Login to save this paper or follow this series

Can We Test for Bias in Scientific Peer-Review?

  • Oswald, Andrew J.


    (University of Warwick)

Science rests upon the reliability of peer review. This paper suggests a way to test for bias. It is able to avoid the fallacy – one seen in the popular press and the research literature – that to measure discrimination it is sufficient to study averages within two populations. The paper’s contribution is primarily methodological, but I apply it, as an illustration, to data from the field of economics. No scientific bias or favoritism is found (although the Journal of Political Economy discriminates against its own Chicago authors). The test’s methodology is applicable in most scholarly disciplines.

If you experience problems downloading a file, check if you have the proper application to view it first. In case of further problems read the IDEAS help page. Note that these files are not on the IDEAS site. Please be patient as the files may be large.

File URL:
Download Restriction: no

Paper provided by Institute for the Study of Labor (IZA) in its series IZA Discussion Papers with number 3665.

in new window

Length: 27 pages
Date of creation: Aug 2008
Date of revision:
Handle: RePEc:iza:izadps:dp3665
Contact details of provider: Postal: IZA, P.O. Box 7240, D-53072 Bonn, Germany
Phone: +49 228 3894 223
Fax: +49 228 3894 180
Web page:

Order Information: Postal: IZA, Margard Ody, P.O. Box 7240, D-53072 Bonn, Germany

References listed on IDEAS
Please report citation or reference errors to , or , if you are the registered author of the cited work, log in to your RePEc Author Service profile, click on "citations" and make appropriate adjustments.:

as in new window
  1. Claudia Goldin & Cecilia Rouse, 1997. "Orchestrating Impartiality: The Impact of "Blind" Auditions on Female Musicians," NBER Working Papers 5903, National Bureau of Economic Research, Inc.
  2. Mirjam van Praag & Bernard M.S. van Praag, 2007. "The Benefits of Being Economics Professor A (and not Z)," CESifo Working Paper Series 1948, CESifo Group Munich.
  3. Eric W. K. Tsang & Bruno S. Frey, 2006. "The as-is journal review process: Let authors own their ideas," IEW - Working Papers 280, Institute for Empirical Research in Economics - University of Zurich.
  4. Frey, Bruno S, 2003. " Publishing as Prostitution?--Choosing between One's Own Ideas and Academic Success," Public Choice, Springer, vol. 116(1-2), pages 205-23, July.
  5. Tom Coupé & Victor Ginsburgh & Abdul Ghafar Noury, 2008. "Are Leading Papers of Better Quality? Evidence from a Natural Experiment," Working Papers ECARES 2008_014, ULB -- Universite Libre de Bruxelles.
  6. Oswald, Andrew J., 2006. "An Examination of the Reliability of Prestigious Scholarly Journals: Evidence and Implications for Decision-makers," IZA Discussion Papers 2070, Institute for the Study of Labor (IZA).
  7. Shamena Anwar & Hanming Fang, 2005. "An Alternative Test of Racial Prejudice in Motor Vehicle Searches: Theory and Evidence," NBER Working Papers 11264, National Bureau of Economic Research, Inc.
  8. J. Peter Neary & James A. Mirrlees & Jean Tirole, 2003. "Evaluating Economics Research in Europe: An Introduction," Journal of the European Economic Association, MIT Press, vol. 1(6), pages 1239-1249, December.
  9. JS Armstrong, 2004. "Peer Review for Journals: Evidence on Quality Control, Fairness, and Innovation," General Economics and Teaching 0412027, EconWPA.
  10. Amanda Goodall, 2005. "Should Research Universities be led by top researchers? Part 1: Are they?," CEE Discussion Papers 0051, Centre for the Economics of Education, LSE.
  11. Justin Wolfers, 2006. "Diagnosing Discrimination: Stock Returns and CEO Gender," NBER Working Papers 11989, National Bureau of Economic Research, Inc.
  12. Daniel S. Hamermesh, 2002. "International Labor Economics," Journal of Labor Economics, University of Chicago Press, vol. 20(4), pages 709-732, October.
  13. Laband, David N & Tollison, Robert D & Karahan, Gokhan R, 2002. "Quality Control in Economics," Kyklos, Wiley Blackwell, vol. 55(3), pages 315-34.
  14. C. Mirjam Van Praag & Bernard M.S. Van Praag, 2008. "The Benefits of Being Economics Professor A (rather than Z)," Economica, London School of Economics and Political Science, vol. 75(300), pages 782-796, November.
  15. Stephen Wu, 2007. "Recent publishing trends at the AER, JPE and QJE," Applied Economics Letters, Taylor & Francis Journals, vol. 14(1), pages 59-63.
  16. Scott Smart & Joel Waldfogel, 1996. "A Citation-Based Test for Discrimination at Economics and Finance Journals," NBER Working Papers 5460, National Bureau of Economic Research, Inc.
  17. Timothy Clark & Mike Wright, 2007. "Reviewing Journal Rankings and Revisiting Peer Reviews: Editorial Perspectives," Journal of Management Studies, Wiley Blackwell, vol. 44(4), pages 612-621, 06.
  18. Liran Einav & Leeat Yariv, 2006. "What's in a Surname? The Effects of Surname Initials on Academic Success," Journal of Economic Perspectives, American Economic Association, vol. 20(1), pages 175-187, Winter.
  19. Glenn Ellison, 2007. "Is Peer Review in Decline?," NBER Working Papers 13272, National Bureau of Economic Research, Inc.
  20. Daniel S. Hamermesh & Peter Schmidt, 2003. "The Determinants of Econometric Society Fellows Elections," Econometrica, Econometric Society, vol. 71(1), pages 399-407, January.
Full references (including those not matched with items on IDEAS)

This item is featured on the following reading lists or Wikipedia pages:

  1. Economic Logic blog

When requesting a correction, please mention this item's handle: RePEc:iza:izadps:dp3665. See general information about how to correct material in RePEc.

For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: (Mark Fallak)

If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

If references are entirely missing, you can add them using this form.

If the full references list an item that is present in RePEc, but the system did not link to it, you can help with this form.

If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your profile, as there may be some citations waiting for confirmation.

Please note that corrections may take a couple of weeks to filter through the various RePEc services.

This information is provided to you by IDEAS at the Research Division of the Federal Reserve Bank of St. Louis using RePEc data.