IDEAS home Printed from https://ideas.repec.org/a/cog/meanco/v14y2026a12494.html

Can AI Gauge Deliberative Quality? Evaluating Deliberation on Social Media With Large Language Models

Author

Listed:
  • Patrick Frey

    (Centre for Media, Communication and Information Research (ZeMKI), University of Bremen, Germany)

  • Smita Bhattacharya

    (Media Research Methods Lab, Leibniz Institute for Media Research)

  • Hans-Bredow-Institute, Germany
  • Gregor Wiedemann

    (Media Research Methods Lab, Leibniz Institute for Media Research)

  • Hans-Bredow-Institute, Germany
  • Cornelius Puschmann

    (Centre for Media, Communication and Information Research (ZeMKI), University of Bremen, Germany)

Abstract

Although social media has become a central arena for deliberative discourse, debates are frequently characterized by incivility. Due to the volume of online discourse, it is challenging to assess when and where interventions, such as community-based counterspeech, are required. Accordingly, automated measurement of deliberative quality is a powerful tool for facilitating constructive online communication. Since deliberation is multidimensional, attempts to monitor online discourse must consider alternative forms of communication to foster inclusive deliberation. Whereas Type I deliberative dimensions, such as rationality and civility, are grounded in Habermasian ideals, Type II deliberation extends this perspective by incorporating alternative communicative forms, including emotionality and storytelling, into a more inclusive assessment of deliberative quality. Addressing the limitations of prior studies that primarily focused on classic Type I deliberative dimensions and tested only a small number of large language models, this study evaluates multiple open-weight models to conduct a nuanced assessment of deliberative quality. We applied the large language model that achieved the best overall balance between predictive performance, model size, and computational efficiency to a dataset of 64,223 comments from German-language climate change discourse on X. Our results demonstrate that large language models can reliably capture complex Type II deliberative dimensions despite class imbalances and subtle linguistic cues that generally hinder automated text classification. Class imbalance often impairs classification performance due to highly skewed label distributions, and Type II dimensions are particularly difficult to detect as they are often signaled by implicit, context-dependent markers. The empirical analysis also reveals distinct communicative patterns: While public speakers predominantly employ civil, rational, constructive, and common-good-oriented communication, other debating users exhibit higher levels of reciprocal engagement, humor, and negative emotions, alongside more frequent incivility. Our study demonstrates that AI-driven measurement can serve as a foundational tool for identifying norm violations, directing counterspeech toward specific debates, promoting inclusive digital public spheres, and observing broader long-term effects of moderation and interventions on the quality of debate.

Suggested Citation

  • Patrick Frey & Smita Bhattacharya & Hans-Bredow-Institute, Germany & Gregor Wiedemann & Hans-Bredow-Institute, Germany & Cornelius Puschmann, 2026. "Can AI Gauge Deliberative Quality? Evaluating Deliberation on Social Media With Large Language Models," Media and Communication, Cogitatio Press, vol. 14.
  • Handle: RePEc:cog:meanco:v14:y:2026:a:12494
    DOI: 10.17645/mac.12494
    as

    Download full text from publisher

    File URL: https://www.cogitatiopress.com/mediaandcommunication/article/view/12494
    Download Restriction: no

    File URL: https://libkey.io/10.17645/mac.12494?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    More about this item

    Keywords

    ;
    ;
    ;
    ;
    ;
    ;
    ;
    ;

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:cog:meanco:v14:y:2026:a:12494. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    We have no bibliographic references for this item. You can help adding them by using this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: António Vieira or IT Department (email available below). General contact details of provider: https://www.cogitatiopress.com .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.