Improving the Performance of Unimodal Dynamic Hand-Gesture Recognition with Multimodal Training

TL;DR

The paper tackles dynamic hand-gesture recognition by enabling unimodal networks to learn from multimodal data during training. It introduces MTUT, where separate 3D-CNNs for each modality are guided by a spatiotemporal semantic alignment (SSA) loss, aligned via correlation matrices and regulated by a focal parameter to prevent negative transfer. The full objective combines standard classification losses with SSA terms, enabling end-to-end training that improves unimodal testing performance and can boost multimodal fusion. Across VIVA, EgoGesture, and NVGesture, MTUT consistently enhances unimodal accuracies and fusion results, offering a practical approach for systems restricted to a single modality at inference while benefiting from multimodal training.

Abstract

We present an efficient approach for leveraging the knowledge from multiple modalities in training unimodal 3D convolutional neural networks (3D-CNNs) for the task of dynamic hand gesture recognition. Instead of explicitly combining multimodal information, which is commonplace in many state-of-the-art methods, we propose a different framework in which we embed the knowledge of multiple modalities in individual networks so that each unimodal network can achieve an improved performance. In particular, we dedicate separate networks per available modality and enforce them to collaborate and learn to develop networks with common semantics and better representations. We introduce a "spatiotemporal semantic alignment" loss (SSA) to align the content of the features from different networks. In addition, we regularize this loss with our proposed "focal regularization parameter" to avoid negative knowledge transfer. Experimental results show that our framework improves the test time recognition accuracy of unimodal networks, and provides the state-of-the-art performance on various dynamic hand gesture recognition datasets.

Figures (7)

• Figure 1: Training and testing schemes of different types of recognition systems. (a) The system is trained and tested with multiple modalities. (b) The system is trained and tested with a single modality. (c) The system leverages the benefits of multimodal training but can be ran as a unimodal system during testing.
• Figure 2: An example of the RGB and optical flow streams from the NVGesture Dataset molchanov2016online. As can be seen, while for the stationary frames RGB provides better representation, optical flow provides better representation for the dynamic frames.
• Figure 3: The value of focal regularization parameter ($\rho^{m,n}$) when $\beta=2$ for different values of classification losses, $\ell^m_{cls}$ and $\ell^n_{cls}$. Proportional to the classification performances of networks $m$ and $n$, this parameter scales the SSA loss to focus on transferring positive knowledge.
• Figure 4: Training network $m$ with the knowledge of network $n$. Training network $m$, is primarily done with respect to its classifier loss ($\ell^m_{cls}$), but comparing with $\ell^n_{cls}$, $\rho^{m,n}$ determines if involving the SSA loss is necessary, and if yes, it regularizes this loss with respect to the difference between the performances of two networks. Note that in the test time, both networks perform independently.
• Figure 5: Sample sequences from different modalities of used datasets. (a) VIVA hand gesture dataset ohn2014hand. (b) NVGesture dataset molchanov2016online. (c) EgoGesture cao2017egocentriczhang2018egogesture. As can be seen, the modalities in VIVA and EgoGesture datasets are well-aligned, while the depth map is not quite aligned with RGB and Optical flow maps in NVGesture.