IDEAS home Printed from https://ideas.repec.org/p/arx/papers/2609.19866.html

Reproducibility is not construct validity: LLM measurement of institutionally situated communication

Author

Listed:
  • Veronika Batzdorfer

    (KIT)

  • Carlo Romano Marcello Alessandro Santagiustina

    (ALMAnaCH, m\'edialab, Sciences Po)

Abstract

High annotation reproducibility does not necessarily imply that an LLM-inferred measure captures the construct it is intended to measure. We test this distinction using a dataset from the European Commission's AI Act consultation, linking structured survey responses to free-text consultation submissions from the same stakeholders. LLM annotations of consultation submissions are highly reproducible (intraclass correlations > 0.99), yet show limited convergence with survey-reported measures of the nominal construct they were intended to approximate. Divergence between survey-and LLM-inferred text-based measures varies systematically across stakeholder groups: business associations express greater concern about AI risks in text-based consultations than in survey responses ({\=g} = +1.0), whereas public authorities and several nonbusiness groups show smaller or negative divergences. Divergences between scores suggest positive spatial autocorrelation across European countries (Moran's I = 0.347, p = 0.036), indicating that stakeholders from neighboring countries tend toward more similar text-based stances towards AI safety concerns. Despite divergence, survey-reported concerns remain strongly associated with support for explainability across all divergence levels. These results demonstrate that LLM annotation reproducibility can coexist with poor construct correspondence and motivate validation procedures that distinguish reproducibility, construct validity, and communication context variation when LLMs are used as measurement instruments.

Suggested Citation

  • Veronika Batzdorfer & Carlo Romano Marcello Alessandro Santagiustina, 2026. "Reproducibility is not construct validity: LLM measurement of institutionally situated communication," Papers 2609.19866, arXiv.org.
  • Handle: RePEc:arx:papers:2609.19866
    as

    Download full text from publisher

    File URL: https://arxiv.org/pdf/2609.19866
    File Function: Latest version
    Download Restriction: no
    ---><---

    Other versions of this item:

    References listed on IDEAS

    as
    1. Adriana Bunea, 2014. "Explaining Interest Groups' Articulation of Policy Preferences in the European Commission's Open Consultations: An Analysis of the Environmental Policy Area," Journal of Common Market Studies, Wiley Blackwell, vol. 52(6), pages 1224-1241, November.
    2. Beyers, Jan & Arras, Sarah, 2020. "Who feeds information to regulators? Stakeholder diversity in European Union regulatory agency consultations," Journal of Public Policy, Cambridge University Press, vol. 40(4), pages 573-598, December.
    3. Blair, Graeme & Coppock, Alexander & Moor, Margaret, 2020. "When to Worry about Sensitivity Bias: A Social Reference Theory and Evidence from 30 Years of List Experiments," American Political Science Review, Cambridge University Press, vol. 114(4), pages 1297-1315, November.
    4. Gorwa, Robert & Lechowski, Grzegorz & Schneiß, Daniel, 2024. "Platform lobbying: Policy influence strategies and the EU's Digital Services Act," EconStor Open Access Articles and Book Chapters, ZBW - Leibniz Information Centre for Economics, vol. 13(2), pages 1-26.
    5. Margaret E. Roberts & Brandon M. Stewart & Richard A. Nielsen, 2020. "Adjusting for Confounding with Text Matching," American Journal of Political Science, John Wiley & Sons, vol. 64(4), pages 887-903, October.
    6. Le Mens, Gaël & Gallego, Aina, 2025. "Positioning Political Texts with Large Language Models by Asking and Averaging - CORRIGENDUM," Political Analysis, Cambridge University Press, vol. 33(3), pages 283-283, July.
    7. Tobias Bach & Marlene Jugl & Dustin Köhler & Kai Wegrich, 2022. "Regulatory agencies, reputational threats, and communicative responses," Regulation & Governance, John Wiley & Sons, vol. 16(4), pages 1042-1057, October.
    8. Victor Chernozhukov & Denis Chetverikov & Mert Demirer & Esther Duflo & Christian Hansen & Whitney Newey & James Robins, 2018. "Double/debiased machine learning for treatment and structural parameters," Econometrics Journal, Royal Economic Society, vol. 21(1), pages 1-68, February.
    9. Le Mens, Gaël & Gallego, Aina, 2025. "Positioning Political Texts with Large Language Models by Asking and Averaging," Political Analysis, Cambridge University Press, vol. 33(3), pages 274-282, July.
    10. Gorwa, Robert & Lechowski, Grzegorz & Schneiß, Daniel, 2024. "Platform lobbying: Policy influence strategies and the EU's Digital Services Act," Internet Policy Review: Journal on Internet Regulation, Alexander von Humboldt Institute for Internet and Society (HIIG), Berlin, vol. 13(2), pages 1-26.
    Full references (including those not matched with items on IDEAS)

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Sallin, Aurelién, 2021. "Estimating returns to special education: combining machine learning and text analysis to address confounding," Economics Working Paper Series 2109, University of St. Gallen, School of Economics and Political Science.
    2. Kevin Rinz, 2026. "Measuring Overtime Exemption Status: A Task-Based Approach," Working Papers 26-15, Federal Reserve Bank of Cleveland.
    3. Ziwen Zu, 2026. "Talking Politics with Artificial Intelligence," Papers 2607.00551, arXiv.org, revised Jul 2026.
    4. Achim Ahrens & Christian B. Hansen & Mark E. Schaffer & Thomas Wiemann, 2025. "Model Averaging and Double Machine Learning," Journal of Applied Econometrics, John Wiley & Sons, Ltd., vol. 40(3), pages 249-269, April.
    5. Aur'elien Sallin, 2021. "Estimating returns to special education: combining machine learning and text analysis to address confounding," Papers 2110.08807, arXiv.org, revised Feb 2022.
    6. Jiaming Zeng & Michael F. Gensheimer & Daniel L. Rubin & Susan Athey & Ross D. Shachter, 2022. "Uncovering interpretable potential confounders in electronic medical records," Nature Communications, Nature, vol. 13(1), pages 1-14, December.
    7. Matti Ylönen, 2026. "Reconceptualising the Brussels Effect," Journal of Common Market Studies, Wiley Blackwell, vol. 64(1), pages 100-124, January.
    8. Achim Ahrens & Christian B. Hansen & Mark E. Schaffer & Thomas Wiemann, 2024. "ddml: Double/debiased machine learning in Stata," Stata Journal, StataCorp LLC, vol. 24(1), pages 3-45, March.
    9. Adriana Bunea & Sergiu Lipcean, 2024. "Understanding patterns of stakeholder participation in public commenting on bureaucratic policymaking: Evidence from the European Union," Regulation & Governance, John Wiley & Sons, vol. 18(3), pages 874-895, July.
    10. Higton, Joe & Tucker, Joshua A & Barrie, Christopher, 2026. "Request routing services undermine the reproducibility of open weight LLMs," SocArXiv sr9hy_v1, Center for Open Science.
    11. Patrick Parschan & Charlott Jakob, 2026. "Computational measurement of political positions: a review of text-based ideal point estimation algorithms," Quality & Quantity: International Journal of Methodology, Springer, vol. 60(2), pages 5663-5690, April.
    12. Leah A. Jacobs & Alec McClean & Zach Branson & Edward Kennedy & Alex Fixler, 2024. "Incremental Propensity Score Effects for Criminology: An Application Assessing the Relationship Between Homelessness, Behavioral Health Problems, and Recidivism," Journal of Quantitative Criminology, Springer, vol. 40(4), pages 707-726, December.
    13. Mao, Minghai & Raiola, Antonio & Yang, Da, 2025. "Double machine learning for Oaxaca-Blinder decomposition," Economics Letters, Elsevier, vol. 255(C).
    14. Asanov, Anastasiya-Mariya & Asanov, Igor & Buenstorf, Guido, 2024. "A low-cost digital first aid tool to reduce psychological distress in refugees: A multi-country randomized controlled trial of self-help online in the first months after the invasion of Ukraine," Social Science & Medicine, Elsevier, vol. 362(C).
    15. Justin Whitehouse & Qizhao Chen & Morgane Austern & Vasilis Syrgkanis, 2025. "Inference on Optimal Policy Values and Other Irregular Functionals via Softmax Smoothing," Papers 2507.11780, arXiv.org, revised Mar 2026.
    16. Nicolaj N. Mühlbach, 2020. "Tree-based Synthetic Control Methods: Consequences of moving the US Embassy," CREATES Research Papers 2020-04, Department of Economics and Business Economics, Aarhus University.
    17. Kyle Colangelo & Ying-Ying Lee, 2019. "Double debiased machine learning nonparametric inference with continuous treatments," CeMMAP working papers CWP72/19, Centre for Microdata Methods and Practice, Institute for Fiscal Studies.
    18. Martin Huber & Eva‐Maria Oeß, 2026. "A Joint Test of Unconfoundedness and Common Trends," Journal of Applied Econometrics, John Wiley & Sons, Ltd., vol. 41(5), pages 684-709, August.
    19. Ruoxuan Xiong & Allison Koenecke & Michael Powell & Zhu Shen & Joshua T. Vogelstein & Susan Athey, 2021. "Federated Causal Inference in Heterogeneous Observational Data," Papers 2107.11732, arXiv.org, revised Apr 2023.
    20. Arne Henningsen & Guy Low & David Wuepper & Tobias Dalhaus & Hugo Storm & Dagim Belay & Stefan Hirsch, 2026. "Estimating Causal Effects With Observational Data: Guidelines for Agricultural and Applied Economists," Journal of Agricultural Economics, Wiley Blackwell, vol. 77(2), pages 356-382, June.

    More about this item

    NEP fields

    This paper has been announced in the following NEP Reports:

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:arx:papers:2609.19866. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: arXiv administrators (email available below). General contact details of provider: https://arxiv.org/ .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.