ActFusion: a Unified Diffusion Model for Action Segmentation and Anticipation

TL;DR

ActFusion introduces a unified diffusion-based model for temporal action segmentation (TAS) and long-term action anticipation (LTA). By employing anticipative masking with learnable mask tokens, the framework learns to segment visible frames while predicting unseen future frames within a single training process, using an ASFormer-based encoder and a dilated-attention decoder. The method achieves state-of-the-art results on TAS and LTA benchmarks (50 Salads, Breakfast, GTEA) and demonstrates bi-directional benefits between the two tasks, while also addressing realistic evaluation by testing without ground-truth future lengths. This work highlights the potential of task-unified diffusion models for temporal action understanding and motivates further exploration of robust, length-agnostic deployment scenarios.

Abstract

Temporal action segmentation and long-term action anticipation are two popular vision tasks for the temporal analysis of actions in videos. Despite apparent relevance and potential complementarity, these two problems have been investigated as separate and distinct tasks. In this work, we tackle these two problems, action segmentation and action anticipation, jointly using a unified diffusion model dubbed ActFusion. The key idea to unification is to train the model to effectively handle both visible and invisible parts of the sequence in an integrated manner; the visible part is for temporal segmentation, and the invisible part is for future anticipation. To this end, we introduce a new anticipative masking strategy during training in which a late part of the video frames is masked as invisible, and learnable tokens replace these frames to learn to predict the invisible future. Experimental results demonstrate the bi-directional benefits between action segmentation and anticipation. ActFusion achieves the state-of-the-art performance across the standard benchmarks of 50 Salads, Breakfast, and GTEA, outperforming task-specific models in both of the two tasks with a single unified model through joint learning.

Figures (9)

• Figure 1: Task-specific models vs. ActFusion (ours). (a) Conventional task-specific models for TAS and LTA. (b) Our unified model ActFusion to address both tasks. (c) Performance comparison across tasks. Tasks-specific models such as DiffAct liu2023diffusion for TAS and FUTR gong2022future for LTA exhibits poor performance on cross-task evaluations. ActFusion outperforms task-specific models on both TAS and LTA, including TempAgg sener2020temporal, which trains separate models for each task. Note that the performance of ActFusion is the evaluation result of a single model through a single training process. The reported performance represents the average of each task in the original paper or evaluated with the official checkpoint (See Sec. \ref{['sec_sup:exp_details']} for details).
• Figure 2: Overall pipeline of ActFusion. During training, we randomly select one of five masking strategies and apply it to input video frames $\bm{F}$, replacing masked regions with learnable tokens to obtain masked features $\bm{F}'$. These features are processed by the encoder $g$ to produce visual embeddings $\bm{E}$, which condition the decoder $h$ to denoise action labels from $\bm{A}^s$ to $\hat{\bm{A}^0}$ at time-step $s$. For inference, we use different masking strategies depending on the task: no masking for TAS and anticipative masking for LTA. The decoder then iteratively denoises action labels following $\bm{\hat{A}}^S \rightarrow \bm{\hat{A}}^{S-\Delta} \rightarrow ... \rightarrow \bm{\hat{A}}^0$ using the DDIM update rule song2020denoising.
• Figure 3: The number of masked clips $N^\mathrm{R}$
• Figure 4: Inference steps of diffusion process
• Figure 5: Qualitative results