Dual-branch differential channel hypergraph convolutional network for human skeleton based action recognition

Graph Convolutional Networks (GCNs) perform well in skeleton action recognition tasks, but their pairwise node connections make it difficult to effectively model high-order dependencies between non-adjacent joints. To address this issue, hypergraph methods have emerged with the aim of capturing complex associations between multiple joints. However, existing methods either rely on static hypergraph structures or fail to fully exploit feature interactions between channels, limiting their ability to adapt to complex action patterns. Therefore, we propose the Dual-Branch Differential Channel Hypergraph Convolutional Network (DBC-HCN), which leverages hypergraphs’ ability to represent a priori non-natural dependencies in skeletal structures. It extracts spatio-temporal topological information and higher-order correlations by integrating static and dynamic hypergraphs, leveraging channel optimization and inter-hypergraph feature interactions. Our network comprises two parallel streams: a Spatio-Temporal Dynamic Hypergraph Convolutional Network (ST-HCN) and a Channel-Differential Hypergraph Convolutional Network (CD-HCN). The Spatio-Temporal Dynamic Hypergraph Convolutional stream is mainly based on the natural topology of the human skeleton, and uses dynamic hypergraphs to model the dependencies of skeletal points in spatio-temporal dimensions, so as to accurately capture the spatio-temporal characteristics of the movements. In contrast, Channel-Differential Hypergraph Convolutional stream focuses on the feature differences between different channels and extracts the characteristics of motion changes between individual skeletal points during action execution to enhance the portrayal of action details. In order to enhance the network’s representational capability, we fuse the dual streams with different action feature representations, so that the Spatio-Temporal Dynamic Hypergraph Convolutional stream and the Channel-Differential Hypergraph Convolutional stream learn from each other’s representations to better enrich the action feature representations. We experiment the model on three datasets, Kinetics-Skeleton 400, NTU RGB + D 60 and NTU RGB + D 120, and the results show that our proposed network is more competitive. The accuracy reaches 96.9% and 92.7% for the cross X-View and X-Sub benchmarks of the NTU RGB + D 60 dataset, respectively. Our code is publicly available at: https://github.com/hhh1234hhh/DBC-HCN.