Recognizing hand use and hand role at home after stroke from egocentric video

Hand function is a central determinant of independence after stroke. Measuring hand use in the home environment is necessary to evaluate the impact of new interventions, and calls for novel wearable technologies. Egocentric video can capture hand-object interactions in context, as well as show how more-affected hands are used during bilateral tasks (for stabilization or manipulation). Automated methods are required to extract this information. The objective of this study was to use artificial intelligence-based computer vision to classify hand use and hand role from egocentric videos recorded at home after stroke. Twenty-one stroke survivors participated in the study. A random forest classifier, a SlowFast neural network, and the Hand Object Detector neural network were applied to identify hand use and hand role at home. Leave-One-Subject-Out-Cross-Validation (LOSOCV) was used to evaluate the performance of the three models. Between-group differences of the models were calculated based on the Mathews correlation coefficient (MCC). For hand use detection, the Hand Object Detector had significantly higher performance than the other models. The macro average MCCs using this model in the LOSOCV were 0.50 ± 0.23 for the more-affected hands and 0.58 ± 0.18 for the less-affected hands. Hand role classification had macro average MCCs in the LOSOCV that were close to zero for all models. Using egocentric video to capture the hand use of stroke survivors at home is technically feasible. Pose estimation to track finger movements may be beneficial to classifying hand roles in the future.Author summary: This study examines the technical feasibility of a home-based hand assessment that may be used in the future for stroke survivors. Wearable sensors for motion detection have been applied for assessment purposes, but because these devices detect only dynamic movements and fail to capture static movements, they do not fully reflect the dexterity that is unique to hands. A wearable (egocentric) camera can be used to record both types of hand movements and provide contextual information, such as what task is being carried out and how the hands carry out the task. This information is essential to understanding hand recovery levels after stroke. Demonstrating that egocentric video analysis can be used to quantify hand movements is important. We analyzed the self-recorded videos from community-dwelling stroke survivors at home using artificial intelligence methods and successfully detected the hand use of more-affected and less-affected hands during typical routines. Therefore, the automated analysis of egocentric video has potential to be further applied in the long-term tracking of hand recovery in the community. As a future assessment tool, the camera can ultimately evaluate the impact of interventions and help to develop new interventions.