DiGIT: Multi-Dilated Gated Encoder and Central-Adjacent Region Integrated Decoder for Temporal Action Detection Transformer

TL;DR

DiGIT tackles fundamental challenges in query-based temporal action detection by addressing redundant multi-scale features and limited temporal context. It introduces the Multi-Dilated Gated Encoder (MDGE) to capture diverse temporal relations within a single-scale encoding and a Central-Adjacent Region Integrated Decoder (CAID) to fuse central and surrounding context via deformable cross-attention. The approach achieves state-of-the-art results on THUMOS14, ActivityNet v1.3, and HACS-Segment, with performance improvements attributed to MDGE's diverse receptive fields and CAID's comprehensive region sampling, while remaining compatible with existing detectors. This work provides a robust, end-to-end TAD paradigm with practical implications for real-world video understanding tasks.

Abstract

In this paper, we examine a key limitation in query-based detectors for temporal action detection (TAD), which arises from their direct adaptation of originally designed architectures for object detection. Despite the effectiveness of the existing models, they struggle to fully address the unique challenges of TAD, such as the redundancy in multi-scale features and the limited ability to capture sufficient temporal context. To address these issues, we propose a multi-dilated gated encoder and central-adjacent region integrated decoder for temporal action detection transformer (DiGIT). Our approach replaces the existing encoder that consists of multi-scale deformable attention and feedforward network with our multi-dilated gated encoder. Our proposed encoder reduces the redundant information caused by multi-level features while maintaining the ability to capture fine-grained and long-range temporal information. Furthermore, we introduce a central-adjacent region integrated decoder that leverages a more comprehensive sampling strategy for deformable cross-attention to capture the essential information. Extensive experiments demonstrate that DiGIT achieves state-of-the-art performance on THUMOS14, ActivityNet v1.3, and HACS-Segment. Code is available at: https://github.com/Dotori-HJ/DiGIT

Figures (8)

• Figure 1: Convergence curves with InternVideo2 InternVideo2 features on THUMOS14 THUMOS14. Our method boosts the previous query-based detectors like TE-TAD TE-TAD and TadTR TadTR.
• Figure 2: Layer-wise CKA similarity comparison between object detection and TAD. The left and right sides are extracted from pre-encoder and post-encoder features, respectively. The 1–5 or 1–6 labels on each axis correspond to the number of multi-scale feature levels used in the respective models.
• Figure 3: Challenges of center-focused sampling in action distinction. Each sequence shows seven evenly sampled frames across the action duration, using examples from THUMOS14 THUMOS14.
• Figure 4: Overview of DiGIT. Our model processes video features through MDGE to capture distinct feature representations utilizing various receptive fields. Subsequently, CAID captures both central and adjacent region information, enhancing action boundary regression and classification. For simplicity, residual connection and layer normalization are omitted.
• Figure 5: Ablation study on MDGE with InternVideo2 features on THUMOS14. The heatmap shows mAP values for different combinations of kernel size and number of dilated convolutions.