Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

MV-GMN: State Space Model for Multi-View Action Recognition

Yuhui Lin, Jiaxuan Lu, Yue Yong, Jiahao Zhang

TL;DR

MV-GMN introduces a novel state-space graph framework (Multi-View Graph Mamba) for efficient multi-modal, multi-view, multi-temporal action recognition. It combines a Bidirectional State Space Block with a GCN-based graph that uses rule-based and KNN edges to fuse features across viewpoints and time, while maintaining linear inference complexity. The approach achieves state-of-the-art results on NTU RGB+D 60/120 and PKU-MMD datasets, with View-Time configurations delivering top-tier accuracy (e.g., NTU-60 cross-subject 98.2%, cross-view 99.7%; NTU-120 cross-subject 97.3%, cross-view 96.4%; PKU-MMD cross-subject 98.0%, cross-view 99.1%). Comparisons and ablations show superior performance over Transformer-based baselines with substantially fewer parameters, demonstrating strong scalability and practicality for resource-limited settings.

Abstract

Recent advancements in multi-view action recognition have largely relied on Transformer-based models. While effective and adaptable, these models often require substantial computational resources, especially in scenarios with multiple views and multiple temporal sequences. Addressing this limitation, this paper introduces the MV-GMN model, a state-space model specifically designed to efficiently aggregate multi-modal data (RGB and skeleton), multi-view perspectives, and multi-temporal information for action recognition with reduced computational complexity. The MV-GMN model employs an innovative Multi-View Graph Mamba network comprising a series of MV-GMN blocks. Each block includes a proposed Bidirectional State Space Block and a GCN module. The Bidirectional State Space Block introduces four scanning strategies, including view-prioritized and time-prioritized approaches. The GCN module leverages rule-based and KNN-based methods to construct the graph network, effectively integrating features from different viewpoints and temporal instances. Demonstrating its efficacy, MV-GMN outperforms the state-of-the-arts on several datasets, achieving notable accuracies of 97.3\% and 96.7\% on the NTU RGB+D 120 dataset in cross-subject and cross-view scenarios, respectively. MV-GMN also surpasses Transformer-based baselines while requiring only linear inference complexity, underscoring the model's ability to reduce computational load and enhance the scalability and applicability of multi-view action recognition technologies.

MV-GMN: State Space Model for Multi-View Action Recognition

TL;DR

MV-GMN introduces a novel state-space graph framework (Multi-View Graph Mamba) for efficient multi-modal, multi-view, multi-temporal action recognition. It combines a Bidirectional State Space Block with a GCN-based graph that uses rule-based and KNN edges to fuse features across viewpoints and time, while maintaining linear inference complexity. The approach achieves state-of-the-art results on NTU RGB+D 60/120 and PKU-MMD datasets, with View-Time configurations delivering top-tier accuracy (e.g., NTU-60 cross-subject 98.2%, cross-view 99.7%; NTU-120 cross-subject 97.3%, cross-view 96.4%; PKU-MMD cross-subject 98.0%, cross-view 99.1%). Comparisons and ablations show superior performance over Transformer-based baselines with substantially fewer parameters, demonstrating strong scalability and practicality for resource-limited settings.

Abstract

Recent advancements in multi-view action recognition have largely relied on Transformer-based models. While effective and adaptable, these models often require substantial computational resources, especially in scenarios with multiple views and multiple temporal sequences. Addressing this limitation, this paper introduces the MV-GMN model, a state-space model specifically designed to efficiently aggregate multi-modal data (RGB and skeleton), multi-view perspectives, and multi-temporal information for action recognition with reduced computational complexity. The MV-GMN model employs an innovative Multi-View Graph Mamba network comprising a series of MV-GMN blocks. Each block includes a proposed Bidirectional State Space Block and a GCN module. The Bidirectional State Space Block introduces four scanning strategies, including view-prioritized and time-prioritized approaches. The GCN module leverages rule-based and KNN-based methods to construct the graph network, effectively integrating features from different viewpoints and temporal instances. Demonstrating its efficacy, MV-GMN outperforms the state-of-the-arts on several datasets, achieving notable accuracies of 97.3\% and 96.7\% on the NTU RGB+D 120 dataset in cross-subject and cross-view scenarios, respectively. MV-GMN also surpasses Transformer-based baselines while requiring only linear inference complexity, underscoring the model's ability to reduce computational load and enhance the scalability and applicability of multi-view action recognition technologies.
Paper Structure (8 sections, 21 equations, 5 figures, 5 tables)

This paper contains 8 sections, 21 equations, 5 figures, 5 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Methodology
  4. Experimental Results
  5. Effectiveness of RGB and Skeleton Fusion
  6. Effectiveness of Temporal and Viewpoint Fusion
  7. Number of KNN Neighbors
  8. Number of MV-GMN Blocks

Figures (5)

  • Figure 1: Overview of the framework. The proposed MV-GMN employs rule-based and KNN-based edges to correlate temporal and viewpoint information, which is then fed into the proposed Multi-View Graph Mamba network. MV-GMN outperforms Transformer-based models while only requiring linear inference complexity for multi-modal, multi-view, multi-temporal sequence action recognition.
  • Figure 2: Overview of the MV-GMN architecture. A) The framework's overall workflow includes Multi-Modal Feature Extraction, Multi-View Graph Mamba Network, and Action Prediction. B) Each MV-GMN Block consists of a Bidirectional State Space Model Block and a GCN block.
  • Figure 3: Different scanning methods, including viewpoint-priority forward and backward scans, as well as temporal-priority forward and backward scans.
  • Figure 4: The left shows the Rule-based edges based on different times and different viewpoints, while the right shows the KNN-based edges.
  • Figure 5: The left shows the effects of different numbers of KNN neighbors, and the right shows the effects of different numbers of MV-GMN Blocks.