IDEAS home Printed from https://ideas.repec.org/a/gam/jftint/v18y2026i7p346-d1979223.html

Credibility Context Improves Political Claim Classification on LIAR and Transfers to LIAR2 NEW

Author

Listed:
  • Bushra Alkomah

    (Department of Computer Science, College of Engineering, University of Idaho, Moscow, ID 83844, USA)

  • Frederick Sheldon

    (Department of Computer Science, College of Engineering, University of Idaho, Moscow, ID 83844, USA)

Abstract

Automated screening of political claims is challenging because many statements are short, underspecified, and labeled with fine-grained truthfulness categories that are difficult to separate using claim text alone. This study examines whether lightweight credibility context, represented by pre-existing speaker fact-checking history counts, can improve transformer-based political claim classification without external evidence retrieval. We evaluate the LIAR benchmark under both its native six-class formulation and a coarser three-class mapping using two strong pretrained encoders, DeBERTa-v3-large and RoBERTa-large. To isolate the effect of credibility context, we vary only one input factor: whether the five LIAR speaker-history counts (pants-fire, false, barely-true, half-true, mostly-true) are appended to the claim and standard metadata as structured text (Hist ON) or omitted (Hist OFF), while keeping the data split, model architecture, training pipeline, and evaluation protocol fixed. All experiments are repeated across five random seeds (7, 13, 42, 123, 2024) and reported as mean ± standard deviation. On LIAR, Hist ON improves macro-F1 and accuracy across both backbones and both label granularities, with the largest gains in the six-class setting where label ambiguity is highest. In the six-class task, macro-F1 increases from 0.3096 to 0.4773 for DeBERTa-v3-large and from 0.3127 to 0.4855 for RoBERTa-large. In the three-class task, the best Hist ON model reaches 0.6241 macro-F1. Because only five seeds are available, the minimum achievable two-sided paired Wilcoxon p -value is 0.0625; therefore, we do not claim conventional α = 0.05 statistical significance and instead report paired mean differences, seed-level gains, and complementary prediction-level reliability analyses from saved test predictions. To assess whether the benefit is limited to the original LIAR test split, we further evaluate the LIAR-trained checkpoints directly on the full non-overlapping LIAR2 NEW test split without additional fine-tuning. This LIAR-to-LIAR2 NEW transfer evaluation shows that Hist ON improves macro-F1 over Hist OFF in all four backbone/granularity settings. The best absolute transferred macro-F1 is achieved by the three-class DeBERTa-v3-large setting (0.6462), whereas the largest Hist ON minus Hist OFF gain occurs in the six-class RoBERTa-large setting (+0.1433 macro-F1). These two values describe different quantities: absolute performance and improvement over the corresponding Hist OFF baseline. We frame the task as political claim classification, or screening, rather than evidence-grounded fact verification: the method uses speaker-level credibility priors and does not retrieve external evidence. The results support speaker-history credibility context as a low-cost, model-agnostic signal for improving claim screening, while the LIAR2 NEW findings should be interpreted as related-benchmark robustness rather than universal out-of-domain generalization.

Suggested Citation

  • Bushra Alkomah & Frederick Sheldon, 2026. "Credibility Context Improves Political Claim Classification on LIAR and Transfers to LIAR2 NEW," Future Internet, MDPI, vol. 18(7), pages 1-27, June.
  • Handle: RePEc:gam:jftint:v:18:y:2026:i:7:p:346-:d:1979223
    as

    Download full text from publisher

    File URL: https://www.mdpi.com/1999-5903/18/7/346/pdf
    Download Restriction: no

    File URL: https://www.mdpi.com/1999-5903/18/7/346/
    Download Restriction: no
    ---><---

    More about this item

    Keywords

    ;
    ;
    ;
    ;
    ;
    ;
    ;
    ;
    ;
    ;

    JEL classification:

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:gam:jftint:v:18:y:2026:i:7:p:346-:d:1979223. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    We have no bibliographic references for this item. You can help adding them by using this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: MDPI Indexing Manager The email address of this maintainer does not seem to be valid anymore. Please ask MDPI Indexing Manager to update the entry or send us the correct address (email available below). General contact details of provider: https://www.mdpi.com .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.