Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Frame Order Matters: A Temporal Sequence-Aware Model for Few-Shot Action Recognition

Bozheng Li, Mushui Liu, Gaoang Wang, Yunlong Yu

Abstract

In this paper, we propose a novel Temporal Sequence-Aware Model (TSAM) for few-shot action recognition (FSAR), which incorporates a sequential perceiver adapter into the pre-training framework, to integrate both the spatial information and the sequential temporal dynamics into the feature embeddings. Different from the existing fine-tuning approaches that capture temporal information by exploring the relationships among all the frames, our perceiver-based adapter recurrently captures the sequential dynamics alongside the timeline, which could perceive the order change. To obtain the discriminative representations for each class, we extend a textual corpus for each class derived from the large language models (LLMs) and enrich the visual prototypes by integrating the contextual semantic information. Besides, We introduce an unbalanced optimal transport strategy for feature matching that mitigates the impact of class-unrelated features, thereby facilitating more effective decision-making. Experimental results on five FSAR datasets demonstrate that our method set a new benchmark, beating the second-best competitors with large margins.

Frame Order Matters: A Temporal Sequence-Aware Model for Few-Shot Action Recognition

Abstract

In this paper, we propose a novel Temporal Sequence-Aware Model (TSAM) for few-shot action recognition (FSAR), which incorporates a sequential perceiver adapter into the pre-training framework, to integrate both the spatial information and the sequential temporal dynamics into the feature embeddings. Different from the existing fine-tuning approaches that capture temporal information by exploring the relationships among all the frames, our perceiver-based adapter recurrently captures the sequential dynamics alongside the timeline, which could perceive the order change. To obtain the discriminative representations for each class, we extend a textual corpus for each class derived from the large language models (LLMs) and enrich the visual prototypes by integrating the contextual semantic information. Besides, We introduce an unbalanced optimal transport strategy for feature matching that mitigates the impact of class-unrelated features, thereby facilitating more effective decision-making. Experimental results on five FSAR datasets demonstrate that our method set a new benchmark, beating the second-best competitors with large margins.
Paper Structure (21 sections, 10 equations, 6 figures, 3 tables)

This paper contains 21 sections, 10 equations, 6 figures, 3 tables.

Table of Contents

  1. Introduction
  2. Related Works
  3. Methodology
  4. Problem Definition
  5. Temporal Sequence-Aware Model
  6. Overall Framework
  7. Perceiver Adapter
  8. Textual Corpus Enhanced Training.
  9. Model Fine-tuning
  10. Unbalanced Optimal Transport Matching
  11. Training Objective
  12. Experiments
  13. Experimental Details
  14. Performance Comparison
  15. Ablation Studies
  16. ...and 6 more sections

Figures (6)

  • Figure 1: Illustration of the semantic changing when rearranging the video frames. The existing model could barely distinguish reversed video due to equally treated frame features while our temporal sequence-aware model could capture the differences between the videos with different frame orders.
  • Figure 2: The framework of our proposed temporal sequence-aware model(TSAM), which consists of a perceiver-based video encoder that captures both the temporal sequential dynamics and spatial information, a textual corpus enhancement module that incorporates the class-related semantics into the feature prototype, and an unbalanced optimal transport matching module that enhances the feature matching.
  • Figure 3: Illustration of (a) Simple Perceiver-based adapter and (b) our Sequential Perceiver-based adapter.
  • Figure 4: Comparison result of our method and the competitor CLIP-FSAR clipfsar under cross-dataset evaluation.
  • Figure 5: Effect of different partial injection numbers
  • ...and 1 more figures