Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Two-Stream temporal transformer for video action classification

Nattapong Kurpukdee, Adrian G. Bors

TL;DR

The paper introduces a two-stream transformer for video action recognition that fuses appearance information from RGB frames with motion information from RAFT-predicted optical flow. By applying a unified transformer encoder with self-attention across both streams, the model learns inter-stream relationships and robust spatio-temporal features. The approach uses lightweight backbones (MViTv1-B, MViTv2-S, Swin-S) pretrained on Kinetics-400 and demonstrates significant improvements over baselines on UCF101, HMDB51, and Something-Something V2, highlighting the effectiveness of cross-stream self-attention for video understanding. This work advances transformer-based video analysis and points to future gains from better optical-flow estimation and data augmentation strategies for broader practical impact.

Abstract

Motion representation plays an important role in video understanding and has many applications including action recognition, robot and autonomous guidance or others. Lately, transformer networks, through their self-attention mechanism capabilities, have proved their efficiency in many applications. In this study, we introduce a new two-stream transformer video classifier, which extracts spatio-temporal information from content and optical flow representing movement information. The proposed model identifies self-attention features across the joint optical flow and temporal frame domain and represents their relationships within the transformer encoder mechanism. The experimental results show that our proposed methodology provides excellent classification results on three well-known video datasets of human activities.

Two-Stream temporal transformer for video action classification

TL;DR

The paper introduces a two-stream transformer for video action recognition that fuses appearance information from RGB frames with motion information from RAFT-predicted optical flow. By applying a unified transformer encoder with self-attention across both streams, the model learns inter-stream relationships and robust spatio-temporal features. The approach uses lightweight backbones (MViTv1-B, MViTv2-S, Swin-S) pretrained on Kinetics-400 and demonstrates significant improvements over baselines on UCF101, HMDB51, and Something-Something V2, highlighting the effectiveness of cross-stream self-attention for video understanding. This work advances transformer-based video analysis and points to future gains from better optical-flow estimation and data augmentation strategies for broader practical impact.

Abstract

Motion representation plays an important role in video understanding and has many applications including action recognition, robot and autonomous guidance or others. Lately, transformer networks, through their self-attention mechanism capabilities, have proved their efficiency in many applications. In this study, we introduce a new two-stream transformer video classifier, which extracts spatio-temporal information from content and optical flow representing movement information. The proposed model identifies self-attention features across the joint optical flow and temporal frame domain and represents their relationships within the transformer encoder mechanism. The experimental results show that our proposed methodology provides excellent classification results on three well-known video datasets of human activities.
Paper Structure (16 sections, 4 equations, 2 figures, 3 tables)

This paper contains 16 sections, 4 equations, 2 figures, 3 tables.

Table of Contents

  1. Introduction
  2. RELATED WORK
  3. Methodology
  4. Optical flow as movement representation
  5. Video Transformer
  6. Two-Stream Fusion and Classification
  7. Experimental results
  8. Datasets and Tasks
  9. Implementation Details
  10. Video classification results
  11. Video classification performance on UCF101 and HMDB51
  12. Video classification performance on SSv2
  13. Discussions
  14. Data augmentaion
  15. Optical flow
  16. ...and 1 more sections

Figures (2)

  • Figure 1: The two-stream temporal transformer. The raw frames and optical flow are fed as inputs into the transformer neural network. The extracted features are then processed by the transformer encoder. Finally, a Multi-Layer Perceptron (MLP) is used to classify the video classes.
  • Figure 2: Examples of frames and their corresponding optical flow prediction, where the optical flow is estimated by the RAFT model teed2020raft on videos from UCF101, HMDB51, and SSv2 dataset. Each direction in the flow is mapped to a RGB color, according to the vector orientation.