Understanding Multimodal Complementarity for Single-Frame Action Anticipation

TL;DR

The paper tackles single-frame action anticipation by augmenting a sole RGB observation with depth geometry and semantic past-action priors. It introduces AAG+, an enhanced framework that combines a robust RGB-D visual encoder, flexible action-history retrieval, and bidirectional cross-attention fusion to predict actions at δ = 1s into the future. Across IKEA-ASM, Meccano, and Assembly101, AAG+ consistently outperforms its predecessor and remains competitive with state-of-the-art video-based methods while maintaining efficiency. The work reveals that the relative value of visual and textual priors is task-dependent, with textual histories dominating in long-horizon, variable tasks, and robust fusion enabling effective single-frame anticipation in practical settings.

Abstract

Human action anticipation is commonly treated as a video understanding problem, implicitly assuming that dense temporal information is required to reason about future actions. In this work, we challenge this assumption by investigating what can be achieved when action anticipation is constrained to a single visual observation. We ask a fundamental question: how much information about the future is already encoded in a single frame, and how can it be effectively exploited? Building on our prior work on Action Anticipation at a Glimpse (AAG), we conduct a systematic investigation of single-frame action anticipation enriched with complementary sources of information. We analyze the contribution of RGB appearance, depth-based geometric cues, and semantic representations of past actions, and investigate how different multimodal fusion strategies, keyframe selection policies and past-action history sources influence anticipation performance. Guided by these findings, we consolidate the most effective design choices into AAG+, a refined single-frame anticipation framework. Despite operating on a single frame, AAG+ consistently improves upon the original AAG and achieves performance comparable to, or exceeding, that of state-of-the-art video-based methods on challenging anticipation benchmarks including IKEA-ASM, Meccano and Assembly101. Our results offer new insights into the limits and potential of single-frame action anticipation, and clarify when dense temporal modeling is necessary and when a carefully selected glimpse is sufficient.

Figures (11)

• Figure 1: Do we need full video for action anticipation, or does a single frame suffice? By fusing a semantic action history with an informative frame, our model matches in performance that of dense video-based approaches. In this work, we also study the roles of various modalities, fusion strategies, and choices of action history source and keyframe selection methods in overall performance.
• Figure 2: Illustration of AAG+. (a) Overview of AAG+ architecture. Following keyframe selection, three modalities are processed. Depth (green) is estimated and colorized prior to feature extraction and fusion with RGB features (blue). Visual fusion is performed via cross-attention, where RGB acts as the primary modality. Visual features are integrated with textual context (orange) extracted from the action history retrieval module through bidirectional cross-attention (purple). (b) Bidirectional Cross-Attention with Gated Fusion. Input features undergo independent cross-attention and are then combined through a gated fusion, where a learned gating vector $g$ adaptively weights the contribution of each modality prior to summation. (c) Overview of methods for Action History Retrieval. We propose two methods for history retrieval: (yellow) prompting a VLM to predict the past actions sequence, or (red) repurposing the AAG+ architecture to perform action recognition on the current frame and storing the results in a queue of size $N$. In both cases, class names are mapped to a latent space via a text encoder to produce individual embeddings $x_{tn}$, which are subsequently fused into the joint representation $X_t$.
• Figure 3: Confusion matrix-style modality breakdown for AAG and AAG+ across datasets. Each sample is categorized by whether the visual branch (RGB-Depth), the text action-history branch, or both produce a correct prediction.
• Figure S.1: Qualitative comparison of depth frames generated by DAv2 (middle) and DA3 (bottom), with the RGB input shown on top for reference. Note how bjects on the table are more clearly distinguished with DAv2 than with DA3.
• Figure S.2: Sample frames from the Meccano dataset illustrating motion blur caused by the egocentric head-mounted camera (left) and the corresponding frames selected by our keyframe selection method (right).