IDEAS home Printed from https://ideas.repec.org/p/arx/papers/2610.10506.html

Validity Without Ground Truth: What Stated-Preference Economics Offers the Evaluation of Language Models

Author

Listed:
  • Daniel Robert Kling Alexander
  • Catherine Louise Kling

Abstract

Many of the questions now put to large language models have no correct answer to score against: what a policy is worth, which option a user should choose, how to weigh competing values. Stated-preference economics has faced this problem for decades. It judges survey responses without knowing the true value, through a framework of validity and related concepts: content, construct, and criterion validity, reliability, incentive compatibility, and consequentiality. We argue that this framework is a general method for evaluating language models, and we set out what each concept means for LLM evaluation. We demonstrate the approach using a published water-quality stated preference economic valuation survey (Vossler et al. 2023) administered to six models. In this economic application, the validity tests take the form of predictions from economic theory: demand should slope down, and willingness to pay should respond to the scope of the good and to income. The tests separate the models sharply. Two older models fail the most basic test at a household income level of \$75,000, and the two newest pass every test of theoretical validity we can score, but diverge on convergent validity. Passing validity tests shows that a model's answers are coherent, not that they are correct.

Suggested Citation

  • Daniel Robert Kling Alexander & Catherine Louise Kling, 2026. "Validity Without Ground Truth: What Stated-Preference Economics Offers the Evaluation of Language Models," Papers 2610.10506, arXiv.org.
    • Handle: RePEc:arx:papers:2610.10506
    as

    Download full text from publisher

    File URL: https://arxiv.org/pdf/2610.10506
    File Function: Latest version
    Download Restriction: no
    ---><---

    References listed on IDEAS

    as
    1. Catherine L. Kling & Daniel J. Phaneuf & Jinhua Zhao, 2012. "From Exxon to BP: Has Some Number Become Better Than No Number?," Journal of Economic Perspectives, American Economic Association, vol. 26(4), pages 3-26, Fall.
    2. Christian A. Vossler & Maurice Doyon & Daniel Rondeau, 2012. "Truth in Consequentiality: Theory and Field Evidence on Discrete Choice Experiments," American Economic Journal: Microeconomics, American Economic Association, vol. 4(4), pages 145-171, November.
    3. Whitehead, John C., 2016. "Plausible responsiveness to scope in contingent valuation," Ecological Economics, Elsevier, vol. 128(C), pages 17-22.
    4. W. Michael Hanemann, 1984. "Welfare Evaluations in Contingent Valuation Experiments with Discrete Responses," American Journal of Agricultural Economics, Agricultural and Applied Economics Association, vol. 66(3), pages 332-341.
    5. Richard Carson & Theodore Groves, 2007. "Incentive and informational properties of preference questions," Environmental & Resource Economics, Springer;European Association of Environmental and Resource Economists, vol. 37(1), pages 181-210, May.
    6. Bengt Kristrom & Pere Riera, 1996. "Is the income elasticity of environmental improvements less than one?," Environmental & Resource Economics, Springer;European Association of Environmental and Resource Economists, vol. 7(1), pages 45-55, January.
    7. Richard T. Carson & Theodore Groves & John A. List, 2014. "Consequentiality: A Theoretical and Experimental Exploration of a Single Binary Choice," Journal of the Association of Environmental and Resource Economists, University of Chicago Press, vol. 1(1), pages 171-207.
    8. Richard C. Bishop & Kevin J. Boyle, 2019. "Reliability and Validity in Nonmarket Valuation," Environmental & Resource Economics, Springer;European Association of Environmental and Resource Economists, vol. 72(2), pages 559-582, February.
    Full references (including those not matched with items on IDEAS)

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Richard C. Bishop & Kevin J. Boyle, 2021. "On Adding-Up as a Validity Criterion for Stated-Preference Studies," Environmental & Resource Economics, Springer;European Association of Environmental and Resource Economists, vol. 80(3), pages 587-601, November.
    2. Christian A. Vossler & Stéphane Bergeron & Maurice Doyon & Daniel Rondeau, 2023. "Revisiting the Gap between the Willingness to Pay and Willingness to Accept for Public Goods," Journal of the Association of Environmental and Resource Economists, University of Chicago Press, vol. 10(2), pages 413-445.
    3. Lloyd-Smith, Patrick, 2026. "Multi-level regression and post-stratification for discrete choice modelling and stated preference research," Journal of Environmental Economics and Management, Elsevier, vol. 138(C).
    4. Robert J. Johnston & Kevin J. Boyle & Wiktor (Vic) Adamowicz & Jeff Bennett & Roy Brouwer & Trudy Ann Cameron & W. Michael Hanemann & Nick Hanley & Mandy Ryan & Riccardo Scarpa & Roger Tourangeau & Ch, 2017. "Contemporary Guidance for Stated Preference Studies," Journal of the Association of Environmental and Resource Economists, University of Chicago Press, vol. 4(2), pages 319-405.
    5. John C. Whitehead & Tim Haas & Lynne Lewis & Leslie Richardson & Pete Schuhmann, 2025. "A Meta-Analysis of SBC Contingent Valuation: Willingness to Pay Estimates, Determinants of Reliability and Replication of Split-Sample Hypothesis Tests," Working Papers 25-11, Department of Economics, Appalachian State University.
    6. Xie, Lusi & Adamowicz, Wiktor & Kecinski, Maik & Fooks, Jacob R., 2022. "Using economic experiments to assess the validity of stated preference contingent behavior responses," Journal of Environmental Economics and Management, Elsevier, vol. 114(C).
    7. John C. Whitehead & Gregory Howard & Louis Cornicelli, 2025. "Total Economic Valuation of Great Lakes Recreational Fisheries: Attribute Nonattendance, Hypothetical Bias, and Insensitivity to Scope," Land Economics, University of Wisconsin Press, vol. 101(4), pages 550-565.
    8. Drichoutis, Andreas C. & Vassilopoulos, Achilleas & Lusk, Jayson L. & Nayga, Rodolfo M. Jr., 2015. "Reference dependence, consequentiality and social desirability in value elicitation: A study of fair labor labeling," 143rd Joint EAAE/AAEA Seminar, March 25-27, 2015, Naples, Italy 202705, European Association of Agricultural Economists.
    9. Bishop, Richard C., 2018. "Warm Glow, Good Feelings, and Contingent Valuation," Journal of Agricultural and Resource Economics, Western Agricultural Economics Association, vol. 43(3), September.
    10. Pengfei Liu & Lingling Hou & Dongqing Li & Shi Min & Yueying Mu, 2021. "Determinants of Livestock Insurance Demand: Experimental Evidence from Chinese Herders," Journal of Agricultural Economics, Wiley Blackwell, vol. 72(2), pages 430-451, June.
    11. Andor, Mark A. & Lange, Andreas & Sommer, Stephan, 2022. "Fairness and the support of redistributive environmental policies," Journal of Environmental Economics and Management, Elsevier, vol. 114(C).
    12. Zawojska, Ewa & Bartczak, Anna & Czajkowski, Mikotaj, "undated". "Disentangling impacts of payment and provision consequentiality and risk attitudes on stated preferences," Annual Meeting, 2017, June 18-21, Montreal, Canada 258602, Canadian Agricultural Economics Society.
    13. Drichoutis, Andreas C. & Vassilopoulos, Achilleas & Lusk, Jayson & Nayga, Rodolfo M., 2015. "Fair farming: Preferences for fair labor certification using four elicitation methods," MPRA Paper 62546, University Library of Munich, Germany.
    14. Zawojska, Ewa & Bartczak, Anna & Czajkowski, Mikołaj, 2019. "Disentangling the effects of policy and payment consequentiality and risk attitudes on stated preferences," Journal of Environmental Economics and Management, Elsevier, vol. 93(C), pages 63-84.
    15. Czajkowski, Mikołaj & Zawojska, Ewa & Meade, Norman & da Motta, Ronaldo Seroa & Welsh, Mike & Ortiz, Ramon Arigoni, 2024. "On the inference about a willingness-to-pay distribution using contingent valuation data," Ecological Economics, Elsevier, vol. 222(C).
    16. Czajkowski, Mikołaj & Vossler, Christian A. & Budziński, Wiktor & Wiśniewska, Aleksandra & Zawojska, Ewa, 2017. "Addressing empirical challenges related to the incentive compatibility of stated preferences methods," Journal of Economic Behavior & Organization, Elsevier, vol. 142(C), pages 47-63.
    17. Abate, Tenaw G. & Börger, Tobias & Aanesen, Margrethe & Falk-Andersson, Jannike & Wyles, Kayleigh J. & Beaumont, Nicola, 2020. "Valuation of marine plastic pollution in the European Arctic: Applying an integrated choice and latent variable model to contingent valuation," Ecological Economics, Elsevier, vol. 169(C).
    18. Manuel Frondel & Stephan Sommer & Lukas Tomberg, 2021. "WTA-WTP Disparity: The Role of Perceived Realism of the Valuation Setting," Land Economics, University of Wisconsin Press, vol. 97(1), pages 196-206.
    19. Haghani, Milad & Bliemer, Michiel C.J. & Rose, John M. & Oppewal, Harmen & Lancsar, Emily, 2021. "Hypothetical bias in stated choice experiments: Part II. Conceptualisation of external validity, sources and explanations of bias and effectiveness of mitigation methods," Journal of choice modelling, Elsevier, vol. 41(C).
    20. Alix-Garcia, Jennifer M. & Sims, Katharine R.E. & Phaneuf, Daniel J., 2019. "Using referenda to improve targeting and decrease costs of conditional cash transfers," Journal of Public Economics, Elsevier, vol. 176(C), pages 179-194.

    More about this item

    NEP fields

    This paper has been announced in the following NEP Reports:

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:arx:papers:2610.10506. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: arXiv administrators (email available below). General contact details of provider: https://arxiv.org/ .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.