Reproducibility is not construct validity: LLM measurement of institutionally situated communication
Author
Abstract
Suggested Citation
Download full text from publisher
Other versions of this item:
- Veronika Batzdorfer & Carlo Romano Marcello Alessandro Santagiustina, 2026. "Reproducibility is not construct validity: LLM measurement of institutionally situated communication [Reproductibilité ne signifie pas validité de construit : l’usage des LLM pour mesurer des communications inscrites dans des contextes institution," Working Papers hal-05745153, HAL.
References listed on IDEAS
- Adriana Bunea, 2014. "Explaining Interest Groups' Articulation of Policy Preferences in the European Commission's Open Consultations: An Analysis of the Environmental Policy Area," Journal of Common Market Studies, Wiley Blackwell, vol. 52(6), pages 1224-1241, November.
- Beyers, Jan & Arras, Sarah, 2020. "Who feeds information to regulators? Stakeholder diversity in European Union regulatory agency consultations," Journal of Public Policy, Cambridge University Press, vol. 40(4), pages 573-598, December.
- Blair, Graeme & Coppock, Alexander & Moor, Margaret, 2020. "When to Worry about Sensitivity Bias: A Social Reference Theory and Evidence from 30 Years of List Experiments," American Political Science Review, Cambridge University Press, vol. 114(4), pages 1297-1315, November.
- Gorwa, Robert & Lechowski, Grzegorz & Schneiß, Daniel, 2024. "Platform lobbying: Policy influence strategies and the EU's Digital Services Act," EconStor Open Access Articles and Book Chapters, ZBW - Leibniz Information Centre for Economics, vol. 13(2), pages 1-26.
- Margaret E. Roberts & Brandon M. Stewart & Richard A. Nielsen, 2020. "Adjusting for Confounding with Text Matching," American Journal of Political Science, John Wiley & Sons, vol. 64(4), pages 887-903, October.
- Le Mens, Gaël & Gallego, Aina, 2025. "Positioning Political Texts with Large Language Models by Asking and Averaging - CORRIGENDUM," Political Analysis, Cambridge University Press, vol. 33(3), pages 283-283, July.
- Tobias Bach & Marlene Jugl & Dustin Köhler & Kai Wegrich, 2022. "Regulatory agencies, reputational threats, and communicative responses," Regulation & Governance, John Wiley & Sons, vol. 16(4), pages 1042-1057, October.
- Victor Chernozhukov & Denis Chetverikov & Mert Demirer & Esther Duflo & Christian Hansen & Whitney Newey & James Robins, 2018.
"Double/debiased machine learning for treatment and structural parameters,"
Econometrics Journal, Royal Economic Society, vol. 21(1), pages 1-68, February.
- Victor Chernozhukov & Denis Chetverikov & Mert Demirer & Esther Duflo & Christian Hansen & Whitney K. Newey & James Robins, 2017. "Double/debiased machine learning for treatment and structural parameters," CeMMAP working papers CWP28/17, Centre for Microdata Methods and Practice, Institute for Fiscal Studies.
- Victor Chernozhukov & Denis Chetverikov & Mert Demirer & Esther Duflo & Christian Hansen & Whitney K. Newey & James Robins, 2017. "Double/debiased machine learning for treatment and structural parameters," CeMMAP working papers 28/17, Institute for Fiscal Studies.
- Victor Chernozhukov & Denis Chetverikov & Mert Demirer & Esther Duflo & Christian Hansen & Whitney Newey & James Robins, 2017. "Double/Debiased Machine Learning for Treatment and Structural Parameters," NBER Working Papers 23564, National Bureau of Economic Research, Inc.
- Le Mens, Gaël & Gallego, Aina, 2025. "Positioning Political Texts with Large Language Models by Asking and Averaging," Political Analysis, Cambridge University Press, vol. 33(3), pages 274-282, July.
- Gorwa, Robert & Lechowski, Grzegorz & Schneiß, Daniel, 2024. "Platform lobbying: Policy influence strategies and the EU's Digital Services Act," Internet Policy Review: Journal on Internet Regulation, Alexander von Humboldt Institute for Internet and Society (HIIG), Berlin, vol. 13(2), pages 1-26.
Most related items
These are the items that most often cite the same works as this one and are cited by the same works as this one.- Sallin, Aurelién, 2021. "Estimating returns to special education: combining machine learning and text analysis to address confounding," Economics Working Paper Series 2109, University of St. Gallen, School of Economics and Political Science.
- Kevin Rinz, 2026. "Measuring Overtime Exemption Status: A Task-Based Approach," Working Papers 26-15, Federal Reserve Bank of Cleveland.
- Ziwen Zu, 2026. "Talking Politics with Artificial Intelligence," Papers 2607.00551, arXiv.org, revised Jul 2026.
- Achim Ahrens & Christian B. Hansen & Mark E. Schaffer & Thomas Wiemann, 2025.
"Model Averaging and Double Machine Learning,"
Journal of Applied Econometrics, John Wiley & Sons, Ltd., vol. 40(3), pages 249-269, April.
- Achim Ahrens & Christian B. Hansen & Mark E. Schaffer & Thomas Wiemann, 2024. "Model Averaging and Double Machine Learning," Papers 2401.01645, arXiv.org, revised Sep 2024.
- Ahrens, Achim & Hansen, Christian B. & Schaffer, Mark E & Wiemann, Thomas, 2024. "Model Averaging and Double Machine Learning," IZA Discussion Papers 16714, IZA Network @ LISER.
- Aur'elien Sallin, 2021. "Estimating returns to special education: combining machine learning and text analysis to address confounding," Papers 2110.08807, arXiv.org, revised Feb 2022.
- Jiaming Zeng & Michael F. Gensheimer & Daniel L. Rubin & Susan Athey & Ross D. Shachter, 2022.
"Uncovering interpretable potential confounders in electronic medical records,"
Nature Communications, Nature, vol. 13(1), pages 1-14, December.
- Zeng, Jiaming & Gensheimer, Michael F. & Rubin, Daniel L. & Athey, Susan & Schachter, Ross D., 2021. "Uncovering Interpretable Potential Confounders in Electronic Medical Records," Research Papers 3950, Stanford University, Graduate School of Business.
- Matti Ylönen, 2026. "Reconceptualising the Brussels Effect," Journal of Common Market Studies, Wiley Blackwell, vol. 64(1), pages 100-124, January.
- Achim Ahrens & Christian B. Hansen & Mark E. Schaffer & Thomas Wiemann, 2024.
"ddml: Double/debiased machine learning in Stata,"
Stata Journal, StataCorp LLC, vol. 24(1), pages 3-45, March.
- Christian B. Hansen & Mark E. Schaffer & Thomas Wiemann & Achim Ahrens, 2022. "ddml: Double/debiased machine learning in Stata," Swiss Stata Conference 2022 02, Stata Users Group.
- Achim Ahrens & Christian B. Hansen & Mark E. Schaffer & Thomas Wiemann, 2023. "ddml: Double/debiased machine learning in Stata," Papers 2301.09397, arXiv.org, revised Jan 2024.
- Ahrens, Achim & Hansen, Christian B. & Schaffer, Mark E & Wiemann, Thomas, 2023. "ddml: Double/Debiased Machine Learning in Stata," IZA Discussion Papers 15963, IZA Network @ LISER.
- Adriana Bunea & Sergiu Lipcean, 2024. "Understanding patterns of stakeholder participation in public commenting on bureaucratic policymaking: Evidence from the European Union," Regulation & Governance, John Wiley & Sons, vol. 18(3), pages 874-895, July.
- Higton, Joe & Tucker, Joshua A & Barrie, Christopher, 2026. "Request routing services undermine the reproducibility of open weight LLMs," SocArXiv sr9hy_v1, Center for Open Science.
- Patrick Parschan & Charlott Jakob, 2026. "Computational measurement of political positions: a review of text-based ideal point estimation algorithms," Quality & Quantity: International Journal of Methodology, Springer, vol. 60(2), pages 5663-5690, April.
- Leah A. Jacobs & Alec McClean & Zach Branson & Edward Kennedy & Alex Fixler, 2024. "Incremental Propensity Score Effects for Criminology: An Application Assessing the Relationship Between Homelessness, Behavioral Health Problems, and Recidivism," Journal of Quantitative Criminology, Springer, vol. 40(4), pages 707-726, December.
- Mao, Minghai & Raiola, Antonio & Yang, Da, 2025. "Double machine learning for Oaxaca-Blinder decomposition," Economics Letters, Elsevier, vol. 255(C).
- Asanov, Anastasiya-Mariya & Asanov, Igor & Buenstorf, Guido, 2024. "A low-cost digital first aid tool to reduce psychological distress in refugees: A multi-country randomized controlled trial of self-help online in the first months after the invasion of Ukraine," Social Science & Medicine, Elsevier, vol. 362(C).
- Justin Whitehouse & Qizhao Chen & Morgane Austern & Vasilis Syrgkanis, 2025. "Inference on Optimal Policy Values and Other Irregular Functionals via Softmax Smoothing," Papers 2507.11780, arXiv.org, revised Mar 2026.
- Nicolaj N. Mühlbach, 2020. "Tree-based Synthetic Control Methods: Consequences of moving the US Embassy," CREATES Research Papers 2020-04, Department of Economics and Business Economics, Aarhus University.
- Kyle Colangelo & Ying-Ying Lee, 2019. "Double debiased machine learning nonparametric inference with continuous treatments," CeMMAP working papers CWP72/19, Centre for Microdata Methods and Practice, Institute for Fiscal Studies.
- Martin Huber & Eva‐Maria Oeß, 2026. "A Joint Test of Unconfoundedness and Common Trends," Journal of Applied Econometrics, John Wiley & Sons, Ltd., vol. 41(5), pages 684-709, August.
- Ruoxuan Xiong & Allison Koenecke & Michael Powell & Zhu Shen & Joshua T. Vogelstein & Susan Athey, 2021.
"Federated Causal Inference in Heterogeneous Observational Data,"
Papers
2107.11732, arXiv.org, revised Apr 2023.
- Xiong, Ruoxuan & Koenecke, Allison & Powell, Michael & Shen, Zhu & Vogelstein, Joshua T. & Athey, Susan, 2021. "Federated Causal Inference in Heterogeneous Observational Data," Research Papers 3990, Stanford University, Graduate School of Business.
- Arne Henningsen & Guy Low & David Wuepper & Tobias Dalhaus & Hugo Storm & Dagim Belay & Stefan Hirsch, 2026.
"Estimating Causal Effects With Observational Data: Guidelines for Agricultural and Applied Economists,"
Journal of Agricultural Economics, Wiley Blackwell, vol. 77(2), pages 356-382, June.
- Arne Henningsen & Guy Low & David Wuepper & Tobias Dalhaus & Hugo Storm & Dagim Belay & Stefan Hirsch, 2024. "Estimating Causal Effects with Observational Data: Guidelines for Agricultural and Applied Economists," IFRO Working Paper 2024/03, University of Copenhagen, Department of Food and Resource Economics.
- Arne Henningsen & Guy Low & David Wuepper & Tobias Dalhaus & Hugo Storm & Dagim Belay & Stefan Hirsch, 2025. "Estimating Causal Effects with Observational Data: Guidelines for Agricultural and Applied Economists," Papers 2508.02310, arXiv.org.
More about this item
NEP fields
This paper has been announced in the following NEP Reports:- NEP-AIN-2026-09-21 (Artificial Intelligence)
Statistics
Access and download statisticsCorrections
All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:arx:papers:2609.19866. See general information about how to correct material in RePEc.
If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.
If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .
If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.
For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: arXiv administrators (email available below). General contact details of provider: https://arxiv.org/ .
Please note that corrections may take a couple of weeks to filter through the various RePEc services.
Printed from https://ideas.repec.org/p/arx/papers/2609.19866.html