Urban traffic evaluation with social media data: A consensus-based LLM negotiation paradigm

Evaluating urban traffic systems from the perspective of travelers is critical for traffic planning and operations management, and social media data has been proven to be a rich and timely source for extracting travel experiences. However, it is challenging to evaluate urban traffic using social media data due to difficulties in obtaining a high-quality dataset that reveals travel experiences and conducting precise evaluations based on the collected posts. With recent advancements in large language models (LLMs), this paper establishes a consensus-based LLM negotiation paradigm to address these challenges. Firstly, we propose an LLM-driven automated data collection and processing pipeline for social media data. Specifically, the understanding and reasoning abilities of LLMs facilitate filtering the obtained raw data with high accuracy. More importantly, LLMs do not require any manual labeling of data or extensive pre-training compared to traditional machine learning methods. As for the evaluation system, we leverage the LLMs to cluster and summarize the key indicators for urban traffic evaluation. Furthermore, we propose a consensus-based LLM negotiation mechanism in which two distinct LLMs serve as the generator and discriminator, respectively. They iteratively evaluate and refine their assessments for each social media post until consensus is reached. In this way, the text data is classified into its associated indicators, with scores assigned based on sentiment analysis. Lastly, we propose the LLM-augmented analytic hierarchy process (AHP), by which the weights of each indicator are determined by LLMs with different role-playing prompts. The effectiveness of the proposed framework is validated in Shanghai and Beijing using social media posts collected from Weibo. The obtained dataset by LLMs is significantly enhanced in both quality and quantity. Moreover, the negotiation mechanism of LLMs can provide not only reliable and robust evaluation results, but also identify key issues behind the evaluation results and offer targeted and precise suggestions. This paper contributes to the use of LLMs in future traffic evaluation and operations.