Replacing or enhancing the human coder? Multiclass classification of policy documents with large language models

Classifying policy documents into policy issue topics has been a long-time effort in political science and communication disciplines. In this work, we use the GPT 3.5, 4, 4o, Gemini 1.0 pro, 1.5 pro and 1.5 Flash models from OpenAI and Google respectively, which are pre-trained instruction-tuned Large Language Models (LLM), to classify congressional bills and hearings into the Comparative Agendas Project’s 21 major policy topics. We propose three use-case scenarios and estimate overall weighted F1 scores ranging from 0.44 to 0.82 depending on scenario and LLM models employed. The three scenarios aim at minimal, moderate, and major human interference, respectively. Our results point towards the insufficiency of complete reliance on instruction tuned LLMs, an increasing accuracy along with the human effort exerted, and a surprisingly high accuracy achieved in the most humanly demanding use-case. Our superior use-case, which combined GPT 4 and Gemini 1.5 Pro achieved 0.82 weighted F1 score on the 83% of the data in which the two models agreed. Benchmarking against Babel, a custom trained algorithm for this use case, Babel presents with a 13 and 16 percentage point higher accuracy. Future research, practical considerations, and implications of future LLM developments are discussed.