Large language models as decision-making agents for autonomous vehicles in dynamic urban scenarios

Urban traffic scenarios exhibit high heterogeneity in road topology, vehicle interactions, and traffic rules. Consequently, autonomous vehicles (AVs) face distinct driving objectives across different scenarios and driving stages. Leveraging exceptional capabilities in semantic understanding, reasoning, and generalization, large language models (LLMs) introduce a novel paradigm for tackling the challenges of diverse scenarios. Therefore, we propose the Urban Scenario Planning Agent (USPAgent), an LLM-based autonomous decision-making framework tailored for three typical scenarios: signalized intersections, ramps, and roundabouts. This framework integrates Urban Scenario Description, Reasoning & Action, Data Collection & Memory, and Evaluation modules, which establishes a complete closed-loop system. It effectively spans the entire process from environmental semantic modeling and LLM chain-of-thought (CoT) reasoning to high-quality data screening and continuous model learning. Specifically, the framework employs the Urban Scenario Description module to extract both dynamic and static environmental information. Combined with designed prompt patterns, the generated scenario prompts guide the LLM to make CoT driving decisions. Concurrently, the Data Collection & Memory module captures decision data in real-time, which is then screened by the Evaluation Module to retain high-quality samples. Finally, the collected data is utilized to finetune the general LLM. Extensive experimental results demonstrate that USPAgent comprehensively outperforms general LLMs in both driving score and route completion rate across the three scenarios, thereby validating the robustness of its driving decisions. Furthermore, ablation studies substantiate the complementary effects on generalization among data from different scenarios.