GazeSense: Deep Learning-Based Multimodal Communicator Using Edge AI

Individuals suffering from severe motor and speech impairments often face significant challenges in communication. Existing assistive communication systems are expensive, cloud-dependent, or require specialized hardware. This paper presents GazeSense, a Deep Learning-Based Multimodal Communication System implemented on an Edge AI platform using Raspberry Pi. The proposed system utilizes gaze direction estimation, blink detection, and facial expression recognition to interpret user intentions in real time. Computer vision techniques using OpenCV, YOLOv8 Nano, and TensorFlow Lite are employed for efficient processing on resource-constrained hardware. The detected gestures are converted into predefined text messages and speech output through an offline Text-to-Speech engine. Experimental results demonstrate low-latency performance, enhanced privacy through local processing, and improved accessibility for individuals with severe disabilities. The system provides an affordable, portable, and reliable communication solution suitable for healthcare and rehabilitation environments.