Real-Time Multimodal Threat Detection Using Fused Face Recognition, Gait Analysis, and Micro-Expression Classification

The current paper outlines a live video surveillance solution based on the combination of face identity, gait patterns, and micro-expressions for estimation of a security threat level. Instead of basing on one input signal only, the proposed framework incorporates biometrics and behavior metrics so as to make the assessment of the degree of a threat more resilient against various changes in body posture, partial occlusions, or delays in inferring the identity. The paper suggests using three pre-trained ResNet-18 models for identification of faces, analyzing gait patterns, and assessing emotions. The outputs of the three models are combined via the use of a simple, yet effective score-level fusion algorithm. In order to make sure that the system is able to work in real-time conditions, its architecture features asynchronous worker threads, IoU-based tracking approach, and a temporal vote-based filter for stabilization of user identities. In implementation, the solution exhibited rapid locking of identities of enrolled users, swift detection of suspicious unknown identities, and consistent performance of around 18-22 FPS on the GPU-enabled setup.