Unsupervised exceptional human action detection from repetition of human assembling tasks using entropy signal clustering

Applying Human Action Recognition (HAR) in manufacturing site to recognize the human assembling tasks, representing as repetitions of human actions, is an emerging research area. However, human unintentional movements or actions beyond the pre-determined tasks are inevitable and cause the difficulty of applying machine learning model for training. Obviously, exhaustively listing all exceptional actions for training a model is not feasible. Therefore, when utilizing a pre-trained model or re-training a model by considering the limited number of exceptional actions leads to the low accuracy of action recognition. To overcome this challenge, in this work, an unsupervised detection framework named Entropy Signal Clustering (ESC) was proposed to detect the exceptional actions from a repetition of actions which are assumed to be basis of assembling tasks. In order to handle frame-basis detection, first, Temporal Self-similarity Matrix (TSM) from the encoder model of RepNet was constructed to represent the similarities of the pair-wise image frames extracted from a video. Second, a calculating process was re-invented to compensate the misleading similarity caused by the padding window on the edges of TSM. Then, an innovative measure of entropy was proposed to create the indication statistics of action similarity. Finally, a clustering-based time-series anomaly detection was proposed to detect the discordance representing the exceptional action in an entropy time-series data against the repetition of actions. To validate the proposed framework, a simulated videos dataset was created. The results show the proposed ESC method is able to obtain accuracies from 85 to 99% and recall of both exceptional (79–100%) and standard of procedure task (92–100%) actions across all demonstration videos comparing with Histogram, Matrix Profile, Local Recurrence Rate based Discord Search (LRRDS), and LRRDS Sampling K-means (LSK) methods.