IDEAS home Printed from https://ideas.repec.org/p/iza/izadps/dp3665.html
   My bibliography  Save this paper

Can We Test for Bias in Scientific Peer-Review?

Author

Listed:
  • Oswald, Andrew J.

    () (University of Warwick)

Abstract

Science rests upon the reliability of peer review. This paper suggests a way to test for bias. It is able to avoid the fallacy – one seen in the popular press and the research literature – that to measure discrimination it is sufficient to study averages within two populations. The paper’s contribution is primarily methodological, but I apply it, as an illustration, to data from the field of economics. No scientific bias or favoritism is found (although the Journal of Political Economy discriminates against its own Chicago authors). The test’s methodology is applicable in most scholarly disciplines.

Suggested Citation

  • Oswald, Andrew J., 2008. "Can We Test for Bias in Scientific Peer-Review?," IZA Discussion Papers 3665, Institute for the Study of Labor (IZA).
  • Handle: RePEc:iza:izadps:dp3665
    as

    Download full text from publisher

    File URL: http://ftp.iza.org/dp3665.pdf
    Download Restriction: no

    References listed on IDEAS

    as
    1. Tom Coupé & Victor Ginsburgh & Abdul Noury, 2010. "Are leading papers of better quality? Evidence from a natural experiment," Oxford Economic Papers, Oxford University Press, vol. 62(1), pages 1-11, January.
    2. Shamena Anwar & Hanming Fang, 2006. "An Alternative Test of Racial Prejudice in Motor Vehicle Searches: Theory and Evidence," American Economic Review, American Economic Association, vol. 96(1), pages 127-151, March.
    3. Glenn Ellison, 2011. "Is Peer Review In Decline?," Economic Inquiry, Western Economic Association International, vol. 49(3), pages 635-657, July.
    4. Eric W. K. Tsang & Bruno S. Frey, 2006. "The as-is journal review process: Let authors own their ideas," CREMA Working Paper Series 2006-09, Center for Research in Economics, Management and the Arts (CREMA).
    5. Timothy Clark & Mike Wright, 2007. "Reviewing Journal Rankings and Revisiting Peer Reviews: Editorial Perspectives," Journal of Management Studies, Wiley Blackwell, vol. 44(4), pages 612-621, June.
    6. van Praag, Mirjam C. & van Praag, Bernard M. S., 2007. "The Benefits of Being Economics Professor A (and not Z)," IZA Discussion Papers 2673, Institute for the Study of Labor (IZA).
    7. Andrew J. Oswald, 2007. "An Examination of the Reliability of Prestigious Scholarly Journals: Evidence and Implications for Decision-Makers," Economica, London School of Economics and Political Science, vol. 74(293), pages 21-31, February.
    8. Amanda H Goodall, 2005. "Should Research Universities be Led by Top Researchers? Part 1: Are they?," HEW 0506003, EconWPA.
    9. J. Peter Neary & James A. Mirrlees & Jean Tirole, 2003. "Evaluating Economics Research in Europe: An Introduction," Journal of the European Economic Association, MIT Press, vol. 1(6), pages 1239-1249, December.
    10. Cecilia Rouse & Claudia Goldin, 2000. "Orchestrating Impartiality: The Impact of "Blind" Auditions on Female Musicians," American Economic Review, American Economic Association, vol. 90(4), pages 715-741, September.
    11. Daniel S. Hamermesh & Peter Schmidt, 2003. "The Determinants of Econometric Society Fellows Elections," Econometrica, Econometric Society, vol. 71(1), pages 399-407, January.
    12. C. Mirjam Van Praag & Bernard M.S. Van Praag, 2008. "The Benefits of Being Economics Professor A (rather than Z)," Economica, London School of Economics and Political Science, vol. 75(300), pages 782-796, November.
    13. Liran Einav & Leeat Yariv, 2006. "What's in a Surname? The Effects of Surname Initials on Academic Success," Journal of Economic Perspectives, American Economic Association, vol. 20(1), pages 175-187, Winter.
    14. Daniel S. Hamermesh, 2002. "International Labor Economics," Journal of Labor Economics, University of Chicago Press, vol. 20(4), pages 709-732, October.
    15. JS Armstrong, 2004. "Peer Review for Journals: Evidence on Quality Control, Fairness, and Innovation," General Economics and Teaching 0412027, EconWPA.
    16. Stephen Wu, 2007. "Recent publishing trends at the AER, JPE and QJE," Applied Economics Letters, Taylor & Francis Journals, vol. 14(1), pages 59-63.
    17. Justin Wolfers, 2006. "Diagnosing Discrimination: Stock Returns and CEO Gender," Journal of the European Economic Association, MIT Press, vol. 4(2-3), pages 531-541, 04-05.
    18. Frey, Bruno S, 2003. "Publishing as Prostitution?--Choosing between One's Own Ideas and Academic Success," Public Choice, Springer, vol. 116(1-2), pages 205-223, July.
    19. Laband, David N & Tollison, Robert D & Karahan, Gokhan R, 2002. "Quality Control in Economics," Kyklos, Wiley Blackwell, vol. 55(3), pages 315-334.
    20. Scott Smart & Joel Waldfogel, 1996. "A Citation-Based Test for Discrimination at Economics and Finance Journals," NBER Working Papers 5460, National Bureau of Economic Research, Inc.
    Full references (including those not matched with items on IDEAS)

    Citations

    Blog mentions

    As found by EconAcademics.org, the blog aggregator for Economics research:
    1. A poor test for scientific bias in peer review
      by Economic Logician in Economic Logic on 2009-03-25 19:30:00

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Ana Rute Cardoso & Paulo Guimarães & Klaus F. Zimmermann, 2010. "Comparing the early research performance of PhD graduates in labor economics in Europe and the USA," Scientometrics, Springer;Akadémiai Kiadó, vol. 84(3), pages 621-637, September.
    2. Carole J. Lee & Cassidy R. Sugimoto & Guo Zhang & Blaise Cronin, 2013. "Bias in peer review," Journal of the Association for Information Science & Technology, Association for Information Science & Technology, vol. 64(1), pages 2-17, January.
    3. Brogaard, Jonathan & Engelberg, Joseph & Parsons, Christopher A., 2014. "Networks and productivity: Causal evidence from editor rotations," Journal of Financial Economics, Elsevier, vol. 111(1), pages 251-270.

    More about this item

    Keywords

    discrimination; citations; science; peer-review system;

    JEL classification:

    • H8 - Public Economics - - Miscellaneous Issues

    NEP fields

    This paper has been announced in the following NEP Reports:

    Lists

    This item is featured on the following reading lists or Wikipedia pages:
    1. Economic Logic blog

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:iza:izadps:dp3665. See general information about how to correct material in RePEc.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: (Mark Fallak). General contact details of provider: http://www.iza.org .

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service hosted by the Research Division of the Federal Reserve Bank of St. Louis . RePEc uses bibliographic data supplied by the respective publishers.