VLM-Nav: Mapless UAV navigation using monocular vision driven by vision-language models

Autonomous vehicles, such as Unmanned Aerial Vehicles (UAVs), have the potential to completely reshape various industries such as parcel delivery, agriculture, surveillance, monitoring, and search-and-rescue missions. Consequently, the demand for safe, cost-effective, and intelligent navigation systems is crucial to ensure reliable performance in complex and dynamic environments. In this study, we propose a novel vision-based UAV navigation method that integrates depthmap estimation with a Vision-Language Model (VLM) for efficient obstacle avoidance and path planning. The system processes RGB images captured by the UAV, transforming them into depth maps using DepthAnything-V2, a powerful zero-shot depth estimator. These depth maps are then analyzed by the VLM, which detects nearby obstacles and plans avoidance maneuvers. We have explored the Gemini-flash and GPT-4o models as VLM in our study. A fully connected network integrates the VLM output with the UAV’s relative heading angle to predict the optimal course of action, enabling the UAV to dynamically navigate complex environments toward its target. The system’s effectiveness is validated through simulations in AirSim using Blocks and the Downtown West environment. The UAV consistently reaches its destination, avoiding obstacles and achieving a near-perfect task completion rate of 0.98. By eliminating the need for costly sensors such as LiDAR and operating without pre-existing maps, our solution provides a cost-efficient, generalizable approach to real-time UAV navigation, especially in unfamiliar or dynamic settings, and highlights emerging trends in autonomous systems research that utilize VLMs.