From Recognition to Prediction: Leveraging Sequence Reasoning for Action Anticipation

TL;DR

The paper addresses action anticipation by exploiting intrinsic correlations among actions. It introduces ARR, a two-stage architecture that first recognizes current actions and then performs sequence reasoning via a causal decoder to predict the next action (NAP), with unsupervised pre-training to leverage video temporal dynamics and reduce labeling needs. Evaluated on EK100, EGTEA Gaze+, and 50salads, ARR achieves state-of-the-art or competitive results across datasets, validating the benefits of modeling action relationships and the effectiveness of the self-supervised pre-training regime. This approach advances practical action anticipation by combining transformer-based recognition with attention-driven reasoning, enabling robust predictions in varied video settings.

Abstract

The action anticipation task refers to predicting what action will happen based on observed videos, which requires the model to have a strong ability to summarize the present and then reason about the future. Experience and common sense suggest that there is a significant correlation between different actions, which provides valuable prior knowledge for the action anticipation task. However, previous methods have not effectively modeled this underlying statistical relationship. To address this issue, we propose a novel end-to-end video modeling architecture that utilizes attention mechanisms, named Anticipation via Recognition and Reasoning (ARR). ARR decomposes the action anticipation task into action recognition and sequence reasoning tasks, and effectively learns the statistical relationship between actions by next action prediction (NAP). In comparison to existing temporal aggregation strategies, ARR is able to extract more effective features from observable videos to make more reasonable predictions. In addition, to address the challenge of relationship modeling that requires extensive training data, we propose an innovative approach for the unsupervised pre-training of the decoder, which leverages the inherent temporal dynamics of video to enhance the reasoning capabilities of the network. Extensive experiments on the Epic-kitchen-100, EGTEA Gaze+, and 50salads datasets demonstrate the efficacy of the proposed methods. The code is available at https://github.com/linuxsino/ARR.

Figures (7)

• Figure 1: Description of action anticipation task. In a long video, observed segments of length $\tau_{o}$ are used to infer actions after an anticipation time of length $\tau_{a}$. “Observation Segment” is used as input, and the output is the action category of “Annotated Action Segment” that we predict, and then calculates the prediction loss with its corresponding annotation. $\tau_{o}$ is not fixed and can be selected according to needs, while $\tau_{a}$ is fixed. Different datasets have different settings.
• Figure 2: The probability of next action. We randomly select an action in EK100 dataset as the current action (e.g., "wash plate"). We count the frequency of the next action, and use the frequency to calculate the probability. Although there are thousands of actions in the dataset, we can see that only some actions (e.g., "put plate" and "wash plate") highly correlated will occur after the given action, and most actions will not appear.
• Figure 3: Comparison of our method with previous methods. While previous methods tend to predict actions directly from a given video, our method learns the correlation among actions to benefit prediction by recognizing actions and then conducting sequence reasoning.
• Figure 4: Overall framework of the ARR. The model consists of two components: on the left is a complete action recognition network that identifies the action corresponding to the current video clip within the action annotation sequence. On the right is a complete sequence reasoning network that predicts the next action corresponding to the current video clip within the action annotation sequence.
• Figure 5: Architectures of AIM and Causal Decoder. (a) AIM uses vanilla ViT vit model to initialize the weight of the blue part. Only the Adapter part of the model is trainable during training. (b) This is a standard decoder architecture with masked self-attention, which guarantees that the output is only related to the input before it, all parameters need to be updated during training.