RefAtomNet++: Advancing Referring Atomic Video Action Recognition using Semantic Retrieval based Multi-Trajectory Mamba

TL;DR

This work tackles Referring Atomic Video Action Recognition (RAVAR) by extending the RefAVA dataset to RefAVA++ and proposing RefAtomNet++, a vision-language model that enhances cross-modal alignment through multi-trajectory semantic-retrieval (Mamba) and multi-hierarchical semantic-aligned cross-attention. The approach constructs semantic trajectories at holistic-sentence, partial-keyword, and scene-attribute levels, aggregates them with a memory-efficient state-space model, and fuses them with spatial-temporal video features to jointly localize the referred person and predict fine-grained actions. Empirical results show RefAtomNet++ achieves new state-of-the-art performance on both RefAVA and RefAVA++, with substantial gains over RefAtomNet and strong baselines, along with notable parameter efficiency and robustness to test-time linguistic variations. The work advances RAVAR by providing a scalable dataset and a powerful cross-modal framework that improves localization accuracy and action reasoning in complex multi-person scenes, enabling more reliable human-centric video understanding applications.

Abstract

Referring Atomic Video Action Recognition (RAVAR) aims to recognize fine-grained, atomic-level actions of a specific person of interest conditioned on natural language descriptions. Distinct from conventional action recognition and detection tasks, RAVAR emphasizes precise language-guided action understanding, which is particularly critical for interactive human action analysis in complex multi-person scenarios. In this work, we extend our previously introduced RefAVA dataset to RefAVA++, which comprises >2.9 million frames and >75.1k annotated persons in total. We benchmark this dataset using baselines from multiple related domains, including atomic action localization, video question answering, and text-video retrieval, as well as our earlier model, RefAtomNet. Although RefAtomNet surpasses other baselines by incorporating agent attention to highlight salient features, its ability to align and retrieve cross-modal information remains limited, leading to suboptimal performance in localizing the target person and predicting fine-grained actions. To overcome the aforementioned limitations, we introduce RefAtomNet++, a novel framework that advances cross-modal token aggregation through a multi-hierarchical semantic-aligned cross-attention mechanism combined with multi-trajectory Mamba modeling at the partial-keyword, scene-attribute, and holistic-sentence levels. In particular, scanning trajectories are constructed by dynamically selecting the nearest visual spatial tokens at each timestep for both partial-keyword and scene-attribute levels. Moreover, we design a multi-hierarchical semantic-aligned cross-attention strategy, enabling more effective aggregation of spatial and temporal tokens across different semantic hierarchies. Experiments show that RefAtomNet++ establishes new state-of-the-art results. The dataset and code are released at https://github.com/KPeng9510/refAVA2.

Figures (4)

• Figure 1: Comparison between the AAL task (left) and the RAVAR task (right) reveals a key difference in approach. Traditional atomic action recognition in multi-person settings is usually framed as an action localization problem: regions of interest for each individual must first be identified, followed by per-person action classification and post-processing to isolate the target subjects. In contrast, RAVAR accepts a textual reference together with a video as input and jointly outputs both the atomic actions and the spatial location of the referred individual. This design underscores RAVAR’s efficiency by leveraging textual guidance; it eliminates much of the manual effort required for subject identification and enables focused analysis of the atomic actions for the individual of interest.
• Figure 2: An overview of the dataset statistics. (a) shows the word cloud of the references from RefAVA, (b) shows the word cloud of the references from RefAVA++, and (c) shows the statistics of the number of annotated persons per sample of RefAVA (blue) and RefAVA++ (red).
• Figure 3: An overview of the RefAtomNet++. We derive holistic-sentence and partial-keyword semantic tokens using the BLIPv2 LLM Encoder li2023blip, where keywords guide semantic retrieval of the most relevant video tokens per frame to form trajectories for state-space modeling. In parallel, DETR provides bounding-box coordinates concatenated with the corresponding categorical semantics, which also retrieve temporal visual tokens for trajectory construction and are regarded as scene-attribute semantic tokens. These partial-keyword and scene-attribute retrieved trajectories are aggregated by an independent Mamba layer to capture long-range dependencies based on the provided scanning orders. The resulting multi-level semantic-retrieved visual tokens are fused with the original video features via a multi-hierarchical semantic-aligned cross-attention mechanism. Finally, spatial and temporal branches jointly deliver localization and fine-grained action predictions.
• Figure 4: Qualitative comparisons of BLIPv2li2023blip, RefAtomNet, and RefAtomNet++ on the RefAVA test set. Each row shows a textual reference, the corresponding ground truth actions, and predictions from the three models. RefAtomNet++ consistently localizes the correct referred individual and predicts fine-grained atomic actions more accurately, while BLIPv2 li2023blip and RefAtomNet often confuse subjects or miss critical actions, especially in cluttered multi-person scenarios.