Using Large Language Models to Assess Equity in U.S. Local Government American Rescue Plans

Problem, research strategy, and findingsPlanners need to assess plans to determine their impact on community outcomes. Reading hundreds of pages of text is a challenge for studies that include a large number of cities. We demonstrate here how large language models (LLMs) can be customized to assess plans. We assessed recovery plans submitted to the U.S. Treasury by June 2024 from 56 large U.S. cities and counties regarding their American Rescue Plan Act (ARPA) investments. ARPA provided flexible funds for cities and counties to pursue equity investments. We used a mixed methods approach with human evaluators compared against the LLM results to ground-truth the results and capture any hallucinations. Our evaluation framework for equity included redistribution, procedural, and conceptual dimensions as well as measures to detect masquerading. We assessed language, action, and processes to see to what extent equity was prioritized in ARPA investment plans. With careful training, the LLM produced objective evaluations despite the diversity across plans. However, consistency across evaluations was a problem. Using LLMs benefits from a mixed methods approach with human evaluation to check accuracy.Takeaway for practiceWith careful training, planning scholars can use LLMs to assess equity in plans. Planners have long called for greater attention to equity, and ARPA offered an unprecedented opportunity through flexible federal funding. ARPA’s flexibility requires a careful assessment of plans to distinguish words and actions. Supervised use of our trained LLM model was able to capture masquerading. We found most plans leaned toward the progressive range regarding equity.