Measuring Online Media Ideology with Large Language Models and "Multi-Cue Classification"

Measuring media ideology is essential for researching media bias, media effects, and various important topics in political science, communication, and other social sciences. However, given journalistic norms of objectivity and the complexity of ideology, measuring media ideology accurately is uniquely challenging. Large language models (LLMs) have become valuable tools in this endeavor. Based on media communication theories, I argue that media ideology is expressed via different cues -- the topic, argument, framing, criticism, and sources of the media content -- and that LLMs often miss these. Standard methods of LLM classification also offer little control, flexibility, and data granularity to researchers. Drawing on insights about computational and quantitative measurement methodologies, I introduce the "Multi-Cue Classification" (MQ-Class) approach. With MQ-Class, an LLM classifies the different ideological cues separately and researchers then apply pre-specified weights and thresholds to combine them into one label per text. I compare standard LLM and MQ-Class methods using two example tasks -- classifying the economic and cultural ideologies of a novel sample of online media articles. Across multiple tests, MQ-Class is more accurate and puts researchers "back in the driver's seat." I conclude by discussing how MQ-Class could be implemented for other classification tasks and data.