Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Introducing Gating and Context into Temporal Action Detection

Aglind Reka, Diana Laura Borza, Dominick Reilly, Michal Balazia, Francois Bremond

TL;DR

This work tackles Temporal Action Detection by arguing that macro-transformer architecture influences performance more than self-attention, and introduces a refined, gated feature extraction pipeline. A Temporal Attention Gating (TAG) layer combines a multi-scale convolution branch, a context branch using boundary frames in cross-attention, and an instant branch to produce richer temporal features within a TriDet-based one-stage framework. Extensive experiments on THUMOS14 and EPIC-KITCHEN 100 demonstrate consistent gains over baselines, with ablations confirming that both gating and context contribute to improved localization, especially at higher IoU thresholds. The proposed method advances accurate, context-aware TAD and suggests broader applicability across architectures and modalities for enhanced action detection in untrimmed videos.

Abstract

Temporal Action Detection (TAD), the task of localizing and classifying actions in untrimmed video, remains challenging due to action overlaps and variable action durations. Recent findings suggest that TAD performance is dependent on the structural design of transformers rather than on the self-attention mechanism. Building on this insight, we propose a refined feature extraction process through lightweight, yet effective operations. First, we employ a local branch that employs parallel convolutions with varying window sizes to capture both fine-grained and coarse-grained temporal features. This branch incorporates a gating mechanism to select the most relevant features. Second, we introduce a context branch that uses boundary frames as key-value pairs to analyze their relationship with the central frame through cross-attention. The proposed method captures temporal dependencies and improves contextual understanding. Evaluations of the gating mechanism and context branch on challenging datasets (THUMOS14 and EPIC-KITCHEN 100) show a consistent improvement over the baseline and existing methods.

Introducing Gating and Context into Temporal Action Detection

TL;DR

This work tackles Temporal Action Detection by arguing that macro-transformer architecture influences performance more than self-attention, and introduces a refined, gated feature extraction pipeline. A Temporal Attention Gating (TAG) layer combines a multi-scale convolution branch, a context branch using boundary frames in cross-attention, and an instant branch to produce richer temporal features within a TriDet-based one-stage framework. Extensive experiments on THUMOS14 and EPIC-KITCHEN 100 demonstrate consistent gains over baselines, with ablations confirming that both gating and context contribute to improved localization, especially at higher IoU thresholds. The proposed method advances accurate, context-aware TAD and suggests broader applicability across architectures and modalities for enhanced action detection in untrimmed videos.

Abstract

Temporal Action Detection (TAD), the task of localizing and classifying actions in untrimmed video, remains challenging due to action overlaps and variable action durations. Recent findings suggest that TAD performance is dependent on the structural design of transformers rather than on the self-attention mechanism. Building on this insight, we propose a refined feature extraction process through lightweight, yet effective operations. First, we employ a local branch that employs parallel convolutions with varying window sizes to capture both fine-grained and coarse-grained temporal features. This branch incorporates a gating mechanism to select the most relevant features. Second, we introduce a context branch that uses boundary frames as key-value pairs to analyze their relationship with the central frame through cross-attention. The proposed method captures temporal dependencies and improves contextual understanding. Evaluations of the gating mechanism and context branch on challenging datasets (THUMOS14 and EPIC-KITCHEN 100) show a consistent improvement over the baseline and existing methods.
Paper Structure (17 sections, 8 equations, 3 figures, 4 tables)

This paper contains 17 sections, 8 equations, 3 figures, 4 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Two stage TAD.
  4. One stage TAD.
  5. Feature selection and gating.
  6. Proposed Method
  7. Method Overview
  8. Convolution Branch
  9. Context Branch
  10. Prediction and Training Setup
  11. Experiments and Results
  12. THUMOS14
  13. EPIC-KITCHENS 100
  14. Ablation Studies
  15. Layer Components
  16. ...and 2 more sections

Figures (3)

  • Figure 1: Overview of the proposed method. (a): Based on TriDet, the model consists of a video feature extractor, a feature pyramid extractor, and a boundary-oriented head for action localization and classification. (b): Structure of the proposed Temporal Attention Gating layer.
  • Figure 2: Convolution branch: The video features are processed by two parallel convolutions, $Conv_w$ and $Conv_{kw}$ with different temporal sizes. Their responses are concatenated and then passed through a gating mechanism, which predicts a scalar parameter $\beta$ used to combine the features through linear interpolation.
  • Figure 3: The context branch employs cross-attention to include context in the convolution's central frame (query) representation using boundary frames (key-values), $t$ is the frame index.