Blinded versus Unblinded Review: A Field Study on the Equity of Peer-Review Processes

Blinded review is often proposed as a solution to inequities in merit-based evaluations. We tested this claim in a high-stakes field experiment comparing single-blind review (reviewer identities withheld) and double-blind review (reviewer and author identities withheld) for 530 academic conference submissions. On reliability, both processes showed moderate reliability, leading to only 40% overlap among top-rated submissions. This low overlap indicates that differences between review systems partly stem from noise in the review system. In terms of fairness, our analysis indicates that single-blind reviews favored more senior coauthors and first authors who were PhD students or research scientists while disfavoring Asian (versus White) first authors. Meanwhile, male first authors received slightly higher ratings in both conditions (especially under double-blind), but the greater the proportion of male coauthors, the lower the review ratings were under double-blind (versus single-blind). On validity, we find that for submissions accepted and presented as a talk, neither review ratings nor author characteristics predicted talk quality, number of questions, or attendance. However, review ratings from both systems similarly predicted judged poster presentation quality (for a subset of submissions) and eventual publication (for all submissions). These findings highlight limitations in both single- and double-blind approaches. Double-blind review is not a cure-all for inequities that might arise via single-blind. Yet, including author identities does not appear to enhance reliability or validity enough to justify the risks this information poses in potentially advantaging a particular individual or group.