IDEAS home Printed from https://ideas.repec.org/p/arx/papers/2503.10990.html
   My bibliography  Save this paper

Statistical Impossibility and Possibility of Aligning LLMs with Human Preferences: From Condorcet Paradox to Nash Equilibrium

Author

Listed:
  • Kaizhao Liu
  • Qi Long
  • Zhekun Shi
  • Weijie J. Su
  • Jiancong Xiao

Abstract

Aligning large language models (LLMs) with diverse human preferences is critical for ensuring fairness and informed outcomes when deploying these models for decision-making. In this paper, we seek to uncover fundamental statistical limits concerning aligning LLMs with human preferences, with a focus on the probabilistic representation of human preferences and the preservation of diverse preferences in aligned LLMs. We first show that human preferences can be represented by a reward model if and only if the preference among LLM-generated responses is free of any Condorcet cycle. Moreover, we prove that Condorcet cycles exist with probability converging to one exponentially fast under a probabilistic preference model, thereby demonstrating the impossibility of fully aligning human preferences using reward-based approaches such as reinforcement learning from human feedback. Next, we explore the conditions under which LLMs would employ mixed strategies -- meaning they do not collapse to a single response -- when aligned in the limit using a non-reward-based approach, such as Nash learning from human feedback (NLHF). We identify a necessary and sufficient condition for mixed strategies: the absence of a response that is preferred over all others by a majority. As a blessing, we prove that this condition holds with high probability under the probabilistic preference model, thereby highlighting the statistical possibility of preserving minority preferences without explicit regularization in aligning LLMs. Finally, we leverage insights from our statistical results to design a novel, computationally efficient algorithm for finding Nash equilibria in aligning LLMs with NLHF. Our experiments show that Llama-3.2-1B, aligned with our algorithm, achieves a win rate of 60.55\% against the base model.

Suggested Citation

  • Kaizhao Liu & Qi Long & Zhekun Shi & Weijie J. Su & Jiancong Xiao, 2025. "Statistical Impossibility and Possibility of Aligning LLMs with Human Preferences: From Condorcet Paradox to Nash Equilibrium," Papers 2503.10990, arXiv.org.
  • Handle: RePEc:arx:papers:2503.10990
    as

    Download full text from publisher

    File URL: http://arxiv.org/pdf/2503.10990
    File Function: Latest version
    Download Restriction: no
    ---><---

    References listed on IDEAS

    as
    1. Peter Duersch & Jörg Oechssler & Burkhard Schipper, 2012. "Pure strategy equilibria in symmetric two-player zero-sum games," International Journal of Game Theory, Springer;Game Theory Society, vol. 41(3), pages 553-564, August.
    2. Kurrild-Klitgaard, Peter, 2001. "An Empirical Example of the Condorcet Paradox of Voting in a Large Electorate," Public Choice, Springer, vol. 107(1-2), pages 135-145, April.
    3. Jean‐Marie Blin, 1973. "Intransitive Social Orderings And The Probability Of The Condorcet Effect," Kyklos, Wiley Blackwell, vol. 26(1), pages 25-35, January.
    4. Kelly, Jerry S, 1974. "Voting Anomalies, the Number of Voters, and the Number of Alternatives," Econometrica, Econometric Society, vol. 42(2), pages 239-251, March.
    5. Wilson, Robert, 1972. "Social choice theory without the Pareto Principle," Journal of Economic Theory, Elsevier, vol. 5(3), pages 478-486, December.
    6. William V. Gehrlein, 2006. "Condorcet’s Paradox," Theory and Decision Library C, Springer, number 978-3-540-33799-7, December.
    Full references (including those not matched with items on IDEAS)

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. William Gehrlein, 2002. "Condorcet's paradox and the likelihood of its occurrence: different perspectives on balanced preferences ," Theory and Decision, Springer, vol. 52(2), pages 171-199, March.
    2. Marc Fleurbaey, 2000. "Choix social : une difficulté et de multiples possibilités," Revue Économique, Programme National Persée, vol. 51(5), pages 1215-1232.
    3. Wesley H. Holliday & Eric Pacuit, 2020. "Arrow’s decisive coalitions," Social Choice and Welfare, Springer;The Society for Social Choice and Welfare, vol. 54(2), pages 463-505, March.
    4. repec:hal:wpspec:info:hdl:2441/10286 is not listed on IDEAS
    5. Juan Candeal, 2013. "Invariance axioms for preferences: applications to social choice theory," Social Choice and Welfare, Springer;The Society for Social Choice and Welfare, vol. 41(3), pages 453-471, September.
    6. Herrade Igersheim, 2005. "Extending Xu's results to Arrow''s Impossibility Theorem," Economics Bulletin, AccessEcon, vol. 4(13), pages 1-6.
    7. Hervé Crès, 2000. "Aggregation of Coarse Preferences," SciencePo Working papers Main hal-01064879, HAL.
    8. Crès, Hervé & Gilboa, Itzhak & Vieille, Nicolas, 2024. "Bureaucracy in quest of feasibility," Journal of Mathematical Economics, Elsevier, vol. 114(C).
    9. Lauwers, Luc, 2000. "Topological social choice," Mathematical Social Sciences, Elsevier, vol. 40(1), pages 1-39, July.
    10. Stensholt, Eivind, 2019. "MMP-elections and the assembly size," Discussion Papers 2019/15, Norwegian School of Economics, Department of Business and Management Science.
    11. Elchanan Mossel & Omer Tamuz, 2012. "Complete characterization of functions satisfying the conditions of Arrow’s theorem," Social Choice and Welfare, Springer;The Society for Social Choice and Welfare, vol. 39(1), pages 127-140, June.
    12. Miller, Michael K., 2009. "Social choice theory without Pareto: The pivotal voter approach," Mathematical Social Sciences, Elsevier, vol. 58(2), pages 251-255, September.
    13. Jean-Marie Blin, 1973. "Majority Voting Rules as Behavior Toward Risk," Discussion Papers 38, Northwestern University, Center for Mathematical Studies in Economics and Management Science.
    14. Harrison-Trainor, Matthew, 2022. "An analysis of random elections with large numbers of voters," Mathematical Social Sciences, Elsevier, vol. 116(C), pages 68-84.
    15. Duersch, Peter & Oechssler, Jörg & Schipper, Burkhard C., 2012. "Unbeatable imitation," Games and Economic Behavior, Elsevier, vol. 76(1), pages 88-96.
    16. Mehmet S. Ismail & Ronald Peeters, 2024. "A connection between von Neumann-Morgenstern expected utility and symmetric potential games," Theory and Decision, Springer, vol. 97(4), pages 707-720, December.
    17. Güth, Werner & Vittoria Levati, M. & Montinari, Natalia, 2014. "Ranking alternatives by a fair bidding rule: A theoretical and experimental analysis," European Journal of Political Economy, Elsevier, vol. 34(C), pages 206-221.
    18. Campbell, Donald E. & Kelly, Jerry S., 2015. "Social choice trade-off results for conditions on triples of alternatives," Mathematical Social Sciences, Elsevier, vol. 77(C), pages 42-45.
    19. Luigi Mittone, 2003. "The Condorcet paradox: an experimental approach to a voting process," CEEL Working Papers 0306, Cognitive and Experimental Economics Laboratory, Department of Economics, University of Trento, Italia.
    20. Sholomov, Lev A., 2000. "Explicit form of neutral social decision rules for basic rationality conditions," Mathematical Social Sciences, Elsevier, vol. 39(1), pages 81-107, January.
    21. Donald E. Campbell & Jerry S. Kelly, 2010. "Uniformly bounded sufficient sets and quasi‐extreme social welfare functions," International Journal of Economic Theory, The International Society for Economic Theory, vol. 6(4), pages 405-412, December.

    More about this item

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:arx:papers:2503.10990. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: arXiv administrators (email available below). General contact details of provider: http://arxiv.org/ .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.