TE-TAD: Towards Full End-to-End Temporal Action Detection via Time-Aligned Coordinate Expression

TL;DR

This work tackles temporal action detection (TAD) in untrimmed videos by addressing the instability and post-processing reliance of existing query-based detectors when handling long video sequences. It introduces TE-TAD, a full end-to-end transformer that uses time-aligned coordinate expressions to convert normalized timeline coordinates into actual video timestamps, coupled with adaptive query selection to match variable video lengths. TE-TAD preserves the set-prediction paradigm while eliminating hand-crafted components like sliding windows and NMS, and employs time-aligned segment refinement across multi-scale features. Empirical results on THUMOS14, ActivityNet v1.3, and EpicKitchens show TE-TAD achieving strong or state-of-the-art performance, with a notable 1) robustness to video length, 2) improvement in stability and convergence, and 3) practical gains from adaptive queries and time-aligned representations for end-to-end TAD.

Abstract

In this paper, we investigate that the normalized coordinate expression is a key factor as reliance on hand-crafted components in query-based detectors for temporal action detection (TAD). Despite significant advancements towards an end-to-end framework in object detection, query-based detectors have been limited in achieving full end-to-end modeling in TAD. To address this issue, we propose \modelname{}, a full end-to-end temporal action detection transformer that integrates time-aligned coordinate expression. We reformulate coordinate expression utilizing actual timeline values, ensuring length-invariant representations from the extremely diverse video duration environment. Furthermore, our proposed adaptive query selection dynamically adjusts the number of queries based on video length, providing a suitable solution for varying video durations compared to a fixed query set. Our approach not only simplifies the TAD process by eliminating the need for hand-crafted components but also significantly improves the performance of query-based detectors. Our TE-TAD outperforms the previous query-based detectors and achieves competitive performance compared to state-of-the-art methods on popular benchmark datasets. Code is available at: https://github.com/Dotori-HJ/TE-TAD

Figures (7)

• Figure 1: Performance comparison of query-based detectors across various feature coverages on THUMOS14, demonstrating how extending the feature coverage impacts detection performance, as measured by mean Average Precision (mAP@AVG). The full end-to-end setting that without coverage constraints is denoted by $\infty$.
• Figure 2: Comparison between sliding window and end-to-end settings. The sliding window generates redundant proposals in the duplicated area.
• Figure 3: Comparative analysis of instability and detection performance on THUMOS14: (a) variance in instance matching quantified by IS; (b) the performance change in mAP. Feature coverage lengths are denoted by the values following the dash (-). The symbol $\infty$ denotes an unrestricted end-to-end setting.
• Figure 4: Analysis of noise tolerance on predicted value. The noise level $\epsilon$ sampled from uniform distribution and injected before the sigmoid function.
• Figure 5: Overview of the TE-TAD. Starting with video input, the architecture processes through a backbone for feature extraction, generating multi-scale features $Z$. These are encoded and subsequently passed through an adaptive query selection, aligning with the video timeline for initial query generation. The decoder refines these queries layer-by-layer, culminating in the refinement of action proposals.