Real-Time Human Fault Detection in Assembly Tasks, Based on Human Action Prediction Using a Spatio-Temporal Learning Model

Human fault detection plays an important role in the industrial assembly process. In the current unstructured industrial workspace, the definition of human faults may vary over a long sequence, and this vagueness introduces multiple issues when using traditional detection methods. A method which could learn the correct action sequence from humans, as well as detect the fault actions based on prior knowledge, would be more appropriate and effective. To this end, we propose an end-to-end learning model to predict future human actions and extend it to detect human faults. We combined the auto-encoder framework and recurrent neural network (RNN) method to predict and generate intuitive future human motions. The convolutional long short-term memory (ConvLSTM) layer was applied to extract spatio-temporal features from video sequences. A score function was implemented to indicate the difference between the correct human action sequence and the fault actions. The proposed model was evaluated on a model vehicle seat assembly task. The experimental results showed that the model could effectively capture the necessary historical details to predict future human actions. The results of several fault scenarios demonstrated that the model could detect the faults in human actions based on corresponding future behaviors through prediction features.