Multimodal AI for Landslide Disaster Response: Fusing IoT Prediction, UAV Detection, and Social Media Triage

Finding a way to predict landslides, detect the physical damage, and organize rescues on the fly is incredibly difficult for emergency teams. This paper presents "Multimodal AI for Landslide Disaster Response: Fusing IoT Prediction, UAV Detection, and Social Media Triage", an end-to-end machine learning framework designed to manage the complete timeline of an emergency. Rather than just sticking to one specific data type, the approach we took breaks disaster management down into three separate phases. To start the pipeline, we apply a Deep Neural Network (DNN) to IoT sensor data so we can forecast landslide risks well before the soil actually gives way. Right after an event hits, a Vision Transformer (ViT-B/16) steps in to look over incoming drone imagery and spot any structural destruction. From there, the system handles the logistics of the aftermath by merging computer vision (ResNet50) with natural language processing (DistilBERT) to read through distress posts that citizens put on social media. To ensure rescue teams are deployed effectively, we incorporated the YOLOv8x algorithm to physically count the number of victims directly from uploaded images. We realized that blindly trusting a neural network's classification is dangerous, so our software actually relies on YOLO's bounding-box tensors to act as a strict safety check. Because of this built-in rule, if a social media post claims there is a critical emergency but the object detector spots zero victims in the photo, the system actively blocks the automated dispatch. This effectively stops AI hallucinations and keeps rescue resources from being wasted. When we ran the evaluations, our sensor prediction model hit 97.8% accuracy, and the drone image classifier came in at 91.42%. These integrated models operate within a single Streamlit dashboard, providing a clear and reliable tool for emergency dispatchers. Future expansions of this work could involve integrating retrieval-augmented generation (RAG) and autonomous AI agents to further optimize rescue routing.