Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

MALT: Multi-scale Action Learning Transformer for Online Action Detection

Zhipeng Yang, Ruoyu Wang, Yang Tan, Liping Xie

TL;DR

MALT addresses online action detection by modeling actions at multiple temporal scales. It combines a hierarchical encoder that produces a coarse-to-fine action feature sequence with a recurrent decoder that fuse multi-scale features efficiently, augmented by a sparse-attention frame scoring mechanism to filter irrelevant history without extra networks. The approach achieves state-of-the-art performance on THUMOS'14 and TVSeries, with ablations confirming the benefits of sparse attention, auxiliary losses, and multi-branch encoding. This multi-scale, attention-based framework offers real-time OAD improvements by preserving local temporal information while reducing computation on uninformative frames. The work also highlights practical considerations, such as encoder depth and filtering thresholds, that influence performance and scalability.

Abstract

Online action detection (OAD) aims to identify ongoing actions from streaming video in real-time, without access to future frames. Since these actions manifest at varying scales of granularity, ranging from coarse to fine, projecting an entire set of action frames to a single latent encoding may result in a lack of local information, necessitating the acquisition of action features across multiple scales. In this paper, we propose a multi-scale action learning transformer (MALT), which includes a novel recurrent decoder (used for feature fusion) that includes fewer parameters and can be trained more efficiently. A hierarchical encoder with multiple encoding branches is further proposed to capture multi-scale action features. The output from the preceding branch is then incrementally input to the subsequent branch as part of a cross-attention calculation. In this way, output features transition from coarse to fine as the branches deepen. We also introduce an explicit frame scoring mechanism employing sparse attention, which filters irrelevant frames more efficiently, without requiring an additional network. The proposed method achieved state-of-the-art performance on two benchmark datasets (THUMOS'14 and TVSeries), outperforming all existing models used for comparison, with an mAP of 0.2% for THUMOS'14 and an mcAP of 0.1% for TVseries.

MALT: Multi-scale Action Learning Transformer for Online Action Detection

TL;DR

MALT addresses online action detection by modeling actions at multiple temporal scales. It combines a hierarchical encoder that produces a coarse-to-fine action feature sequence with a recurrent decoder that fuse multi-scale features efficiently, augmented by a sparse-attention frame scoring mechanism to filter irrelevant history without extra networks. The approach achieves state-of-the-art performance on THUMOS'14 and TVSeries, with ablations confirming the benefits of sparse attention, auxiliary losses, and multi-branch encoding. This multi-scale, attention-based framework offers real-time OAD improvements by preserving local temporal information while reducing computation on uninformative frames. The work also highlights practical considerations, such as encoder depth and filtering thresholds, that influence performance and scalability.

Abstract

Online action detection (OAD) aims to identify ongoing actions from streaming video in real-time, without access to future frames. Since these actions manifest at varying scales of granularity, ranging from coarse to fine, projecting an entire set of action frames to a single latent encoding may result in a lack of local information, necessitating the acquisition of action features across multiple scales. In this paper, we propose a multi-scale action learning transformer (MALT), which includes a novel recurrent decoder (used for feature fusion) that includes fewer parameters and can be trained more efficiently. A hierarchical encoder with multiple encoding branches is further proposed to capture multi-scale action features. The output from the preceding branch is then incrementally input to the subsequent branch as part of a cross-attention calculation. In this way, output features transition from coarse to fine as the branches deepen. We also introduce an explicit frame scoring mechanism employing sparse attention, which filters irrelevant frames more efficiently, without requiring an additional network. The proposed method achieved state-of-the-art performance on two benchmark datasets (THUMOS'14 and TVSeries), outperforming all existing models used for comparison, with an mAP of 0.2% for THUMOS'14 and an mcAP of 0.1% for TVseries.
Paper Structure (15 sections, 13 equations, 3 figures, 6 tables)

This paper contains 15 sections, 13 equations, 3 figures, 6 tables.

Table of Contents

  1. Introduction
  2. RELATED WORK
  3. METHODOLOGY
  4. Problem Definition
  5. Sparse Attention
  6. Hierarchical Encoder
  7. Recurrent Decoder
  8. MALT Training
  9. EXPERIMENTS
  10. Datasets
  11. Implementation Details
  12. Quantitative Comparisons
  13. Ablation Studies
  14. Design Choices
  15. CONCLUSION

Figures (3)

  • Figure 1: An illustration of sparse attention. A matrix was used to represent the attention map between keys and queries. Sparse attention only selects the top-k largest values from each row, which can be used to explicitly score and filter historical frames. For example, $k=3$ was chosen in this figure, such that only the top-3 largest values were selected in each row.
  • Figure 2: An example of OAD. Nine-ball includes a breaking action that consists of a pre-shot routine, cuing of the ball, and striking. Projecting the entire set of action frames to a single latent encoding may lead to a lack of local information, since encoder units can only learn weights for all action frames and not for each specific component, producing only a rough image of the breaking action.
  • Figure 3: An overview of MALT. The hierarchical encoder structure composed of multiple composite encoding branches, used to calculate the cross-attention between other branches. This process encodes historical frames $M_L$ into an action feature sequence $\{g_1,g_2,...,g_N\}$, including a series at different scales. A recurrent decoder then fuses multiple features in a stage-by-stage fashion.