Urban function recognition by integrating social media and street-level imagery

Recognizing urban functions is crucial for understanding urban spatial structures and urban planning. Previous work has investigated urban functions based on human activities that were derived from mobile phone positioning data, check-in data, taxi data, etc. However, urban functions can only be comprehensively sensed from both human activities and the physical environment together. To do so, a deep learning method was proposed to predict urban functions by integrating social media data and street-level imagery. The verbs extracted from social media posts were taken as the proxy for human activities, and we identified urban physical environmental information from street-level imagery. Then urban functions were uncovered from both the verbs in terms of human activities and street-level imagery from the perspective of the physical environment. Twelve types of urban function were recognized by verbs in social media posts, which were then improved by integrating street-level imagery within the 5th Ring Road of Beijing, China. The experiment demonstrated that verbs as direct proxies for human activities can avoid noise, and the multi-source data integration eliminated biases caused by a single data source. This work provides a comprehensive understanding of urban structure and dynamics for urban management and planning.