Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Referring Atomic Video Action Recognition

Kunyu Peng, Jia Fu, Kailun Yang, Di Wen, Yufan Chen, Ruiping Liu, Junwei Zheng, Jiaming Zhang, M. Saquib Sarfraz, Rainer Stiefelhagen, Alina Roitberg

TL;DR

This paper introduces Referring Atomic Video Action Recognition (RAVAR), a task to identify atomic actions of a person specified by a natural language description within a video. It introduces the RefAVA dataset, built on AVA with 17,946 clips and 36,630 textual referring instances covering 80 atomic action categories, and proposes RefAtomNet, a three-stream multimodal architecture that employs location-semantic tokens and cross-stream agent attention to align the referring expression with the target individual and predict the action. The approach demonstrates state-of-the-art performance on RefAVA, outperforming diverse baselines from atomic action localization, VQA, and VTR, and shows robustness to test-time perturbations and generalization across backbones and detectors. This work advances practical, text-guided action understanding in multi-person videos, with potential applications in assistive technologies and human-robot interaction, while acknowledging limitations and directions for handling more complex referential scenarios.

Abstract

We introduce a new task called Referring Atomic Video Action Recognition (RAVAR), aimed at identifying atomic actions of a particular person based on a textual description and the video data of this person. This task differs from traditional action recognition and localization, where predictions are delivered for all present individuals. In contrast, we focus on recognizing the correct atomic action of a specific individual, guided by text. To explore this task, we present the RefAVA dataset, containing 36,630 instances with manually annotated textual descriptions of the individuals. To establish a strong initial benchmark, we implement and validate baselines from various domains, e.g., atomic action localization, video question answering, and text-video retrieval. Since these existing methods underperform on RAVAR, we introduce RefAtomNet -- a novel cross-stream attention-driven method specialized for the unique challenges of RAVAR: the need to interpret a textual referring expression for the targeted individual, utilize this reference to guide the spatial localization and harvest the prediction of the atomic actions for the referring person. The key ingredients are: (1) a multi-stream architecture that connects video, text, and a new location-semantic stream, and (2) cross-stream agent attention fusion and agent token fusion which amplify the most relevant information across these streams and consistently surpasses standard attention-based fusion on RAVAR. Extensive experiments demonstrate the effectiveness of RefAtomNet and its building blocks for recognizing the action of the described individual. The dataset and code will be made publicly available at https://github.com/KPeng9510/RAVAR.

Referring Atomic Video Action Recognition

TL;DR

This paper introduces Referring Atomic Video Action Recognition (RAVAR), a task to identify atomic actions of a person specified by a natural language description within a video. It introduces the RefAVA dataset, built on AVA with 17,946 clips and 36,630 textual referring instances covering 80 atomic action categories, and proposes RefAtomNet, a three-stream multimodal architecture that employs location-semantic tokens and cross-stream agent attention to align the referring expression with the target individual and predict the action. The approach demonstrates state-of-the-art performance on RefAVA, outperforming diverse baselines from atomic action localization, VQA, and VTR, and shows robustness to test-time perturbations and generalization across backbones and detectors. This work advances practical, text-guided action understanding in multi-person videos, with potential applications in assistive technologies and human-robot interaction, while acknowledging limitations and directions for handling more complex referential scenarios.

Abstract

We introduce a new task called Referring Atomic Video Action Recognition (RAVAR), aimed at identifying atomic actions of a particular person based on a textual description and the video data of this person. This task differs from traditional action recognition and localization, where predictions are delivered for all present individuals. In contrast, we focus on recognizing the correct atomic action of a specific individual, guided by text. To explore this task, we present the RefAVA dataset, containing 36,630 instances with manually annotated textual descriptions of the individuals. To establish a strong initial benchmark, we implement and validate baselines from various domains, e.g., atomic action localization, video question answering, and text-video retrieval. Since these existing methods underperform on RAVAR, we introduce RefAtomNet -- a novel cross-stream attention-driven method specialized for the unique challenges of RAVAR: the need to interpret a textual referring expression for the targeted individual, utilize this reference to guide the spatial localization and harvest the prediction of the atomic actions for the referring person. The key ingredients are: (1) a multi-stream architecture that connects video, text, and a new location-semantic stream, and (2) cross-stream agent attention fusion and agent token fusion which amplify the most relevant information across these streams and consistently surpasses standard attention-based fusion on RAVAR. Extensive experiments demonstrate the effectiveness of RefAtomNet and its building blocks for recognizing the action of the described individual. The dataset and code will be made publicly available at https://github.com/KPeng9510/RAVAR.
Paper Structure (36 sections, 14 equations, 8 figures, 15 tables)

This paper contains 36 sections, 14 equations, 8 figures, 15 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. RAVAR: Established Benchmark
  4. Introduction of the RefAVA Dataset
  5. The Baselines for the RAVAR Benchmark
  6. RefAtomNet: Proposed Method
  7. Overview
  8. RefAtomNet
  9. Loss Functions
  10. Experiments
  11. Implementation Details
  12. Experimental Results
  13. Conclusions
  14. Potential Impacts
  15. Discussion of Limitations
  16. ...and 21 more sections

Figures (8)

  • Figure 1: A comparison of AAL Task (left) and RAVAR task (right).
  • Figure 2: Chord visualization of several keywords.
  • Figure 3: Annotation statistics: #The number of annotated persons contained in the scenario (x-axis); #The number of instances (y-axis).
  • Figure 5: An overview of the RefAtomNet architecture.
  • Figure 6: An overview of qualitative results. Missed predictions are marked with a red cross, while true positive and false positive predictions are shown in green and red.
  • ...and 3 more figures