Systematic Evaluation of Nation-State Propaganda in LLM Outputs

The rapid growth and usability of AI is creating new risks to information quality, including the possibility that chatbot responses spread propaganda from state-backed influence campaigns. There is little systematic evidence regarding how frequently LLMs spread such narratives when responding to user queries. Previous attempts to assess these risks rely on small numbers of evaluation questions, use opaque query construction procedures, and fail to account for key sources of uncertainty in model responses. We address these gaps with an automated, reproducible pipeline that samples documented state-backed propaganda and news items, programmatically generates evaluation questions, and audits LLM-generated responses. We evaluate whether five prominent models: (1) reproduce arguments aligned with documented Kremlin propaganda narratives; (2) present the Kremlin perspective on events without providing corrective context; and (3) cite known propaganda outlets. We find substantially lower rates of propaganda-aligned responses and propaganda-source citation than prior small-sample work would suggest. These risks are particularly rare for questions resembling ordinary information searches about Ukraine. Our approach enables systematic comparisons across models and issue areas, supports ongoing monitoring, and could be extended to many other hotly-debated topics.