IDEAS home Printed from https://ideas.repec.org/p/cue/wpaper/awp-08-2026.html

Fairness Trade-offs in Expert Evaluation: Information, Judgment, and Procedural Design

Author

Listed:
  • Karol J. Borowiecki

    (University of Southern Denmark)

  • Marc T. Law

    (University of Vermont)

Abstract

Evaluation systems often seek to reduce bias, prevent manipulation, preserve relevant information, maintain comparability, and treat candidates equally, but these fairness goals can conflict. This paper develops a framework for understanding such trade-offs in expert evaluation—including peer review, grant panels, academic hiring, and artistic competitions—where quality is multidimensional and legitimate disagreement among evaluators is possible. We treat the choice among evaluation procedures as a problem of constitutional design: rules must be selected before the specific candidates, evaluators, and circumstances that will test them are known. Classical music competitions are our primary analytical setting because their rules, scores, and controversies make these trade-offs unusually visible, but the framework applies to other settings in which expert judgments are aggregated under incomplete information. The framework highlights two core tensions: information control and outlier rules. First, when information is both quality-relevant and identity-revealing, procedures that suppress bias-relevant cues may also remove information with genuine evaluative content. Second, because sincere minority judgment and strategic manipulation can generate similar observed score patterns, score-based exclusion rules may reduce manipulation while limiting legitimate dissent. Two further trade-offs concerning transparency and standardization are developed as illustrations. The analysis shows that no procedure can eliminate all vulnerabilities at once. Beyond this design implication, the framework also helps explain why evaluation rules differ across stages and settings, and why well-intentioned reforms may create new vulnerabilities.

Suggested Citation

  • Karol J. Borowiecki & Marc T. Law, 2026. "Fairness Trade-offs in Expert Evaluation: Information, Judgment, and Procedural Design," ACEI Working Paper Series AWP-08-2026, Association for Cultural Economics International.
  • Handle: RePEc:cue:wpaper:awp-08-2026
    as

    Download full text from publisher

    File URL: http://files.culturaleconomics.org/papers/AWP-08-2026.pdf
    Download Restriction: no
    ---><---

    More about this item

    Keywords

    ;
    ;
    ;
    ;
    ;
    ;
    ;

    JEL classification:

    • D02 - Microeconomics - - General - - - Institutions: Design, Formation, Operations, and Impact
    • D63 - Microeconomics - - Welfare Economics - - - Equity, Justice, Inequality, and Other Normative Criteria and Measurement
    • D71 - Microeconomics - - Analysis of Collective Decision-Making - - - Social Choice; Clubs; Committees; Associations
    • D72 - Microeconomics - - Analysis of Collective Decision-Making - - - Political Processes: Rent-seeking, Lobbying, Elections, Legislatures, and Voting Behavior
    • D82 - Microeconomics - - Information, Knowledge, and Uncertainty - - - Asymmetric and Private Information; Mechanism Design
    • Z11 - Other Special Topics - - Cultural Economics - - - Economics of the Arts and Literature

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:cue:wpaper:awp-08-2026. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    We have no bibliographic references for this item. You can help adding them by using this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Paul Crosby (email available below). General contact details of provider: https://edirc.repec.org/data/aceiiea.html .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.