Action-slot: Visual Action-centric Representations for Multi-label Atomic Activity Recognition in Traffic Scenes

TL;DR

The paper tackles multi-label atomic activity recognition in traffic scenes, introducing Action-slot, a slot-attention framework that learns action-centric representations without explicit perception guidance. It allocates a fixed number of action slots (one per activity class) plus a background slot, and updates them in parallel over spatial-temporal tokens, with a negative-class regularization to sharpen focus. To address data imbalance in real-world datasets, the authors create the synthetic TACO dataset via CARLA, enabling balanced coverage of 64 activity classes and effective pretraining that transfers to OATS and nuScenes. Experiments show Action-slot achieving state-of-the-art results on OATS, substantial gains on TACO, and improved real-world performance when pretrained on TACO, highlighting its robustness and transferability for traffic scenario understanding.

Abstract

In this paper, we study multi-label atomic activity recognition. Despite the notable progress in action recognition, it is still challenging to recognize atomic activities due to a deficiency in a holistic understanding of both multiple road users' motions and their contextual information. In this paper, we introduce Action-slot, a slot attention-based approach that learns visual action-centric representations, capturing both motion and contextual information. Our key idea is to design action slots that are capable of paying attention to regions where atomic activities occur, without the need for explicit perception guidance. To further enhance slot attention, we introduce a background slot that competes with action slots, aiding the training process in avoiding unnecessary focus on background regions devoid of activities. Yet, the imbalanced class distribution in the existing dataset hampers the assessment of rare activities. To address the limitation, we collect a synthetic dataset called TACO, which is four times larger than OATS and features a balanced distribution of atomic activities. To validate the effectiveness of our method, we conduct comprehensive experiments and ablation studies against various action recognition baselines. We also show that the performance of multi-label atomic activity recognition on real-world datasets can be improved by pretraining representations on TACO. We will release our source code and dataset. See the videos of visualization on the project page: https://hcis-lab.github.io/Action-slot/

Figures (14)

• Figure 1: Illustration of the concept of multi-label atomic activity recognition and our proposed Action-slot. In the scene, three atomic activities are presented and depicted by colored arrows. For example, the red arrow represents the Z1-Z4: C+ atomic activity, indicating a group of vehicles turning left. Atomic activities are defined based on road user's type and their motion patterns grounded in the underlying road structure. We introduce Action-slot to learn visual action-centric representations that enable decomposing multiple atomic activities in videos. We demonstrate that our framework can effectively recognize multiple atomic activities via learned representations.
• Figure 2: The distribution of atomic activity classes in our TACO dataset compared to the OATS dataset. Please note that, for space considerations, we omit the road topology notations of corner (C) and roadway (Z) on the x-axis of the figure.
• Figure 3: The top of the figure illustrates the proposed framework. Action-slot takes video as input and uses a CNN encoder to extract feature patches. All patches are then processed with individual slots simultaneously to find the most relevant spatial-temporal patches corresponding to each action slot. The updated action slots are fed into a fully connected layer to predict the probability of the corresponding action class, excluding the background slot. The bottom of the figure depicts the attention maps of action and background slots. We propose to incorporate a background mask $M_{\mathtt{bg}}$ to supervise the background slot. The design facilitates other action slots to capture action signals. Furthermore, we design a regularization for slots allocated to negative classes (e.g., Z3-Z4: C) using an all-zero mask $M_{\mathtt{neg}}$.
• Figure 4: (a) Our parallel scheme considers updating slots based on the spatial-temporal features for all the frames. (b) The slots are updated recurrently along the temporal dimension, where each time the slot only considers a frame-wise feature.
• Figure 5: Visualization of attention maps learned from OATS. Colored masks represent the attention of the activity slots Z4-Z3:C and C2-C1:P+. Note that, while MO successfully predicts the activity C2-C1:P+, the corresponding attention scores are very low.