Anticipating Object State Changes in Long Procedural Videos

TL;DR

The paper defines Object State Change Anticipation (OSCA), a new task to predict the forthcoming object state change at the start of the next unobserved action in long procedural videos. It introduces a multimodal framework that fuses recent visual cues with a textual history of past actions and object states, and grounds evaluation on the Ego4D-OSCA extension of Ego4D. The authors demonstrate that incorporating lexical histories substantially improves near-future state prediction, with larger gains under oracle recognizers and robust behavior even with noisy recognizers, highlighting the value of video-language integration for proactive video understanding and task planning. The Ego4D-OSCA dataset provides a challenging benchmark of long, multi-action egocentric videos with nine state-change classes (including No OSC) and inverse-state pairs, enabling broader future work, including leveraging LLMs and zero-shot settings for unseen objects or actions.

Abstract

In this work, we introduce (a) the new problem of anticipating object state changes in images and videos during procedural activities, (b) new curated annotation data for object state change classification based on the Ego4D dataset, and (c) the first method for addressing this challenging problem. Solutions to this new task have important implications in vision-based scene understanding, automated monitoring systems, and action planning. The proposed novel framework predicts object state changes that will occur in the near future due to yet unseen human actions by integrating learned visual features that represent recent visual information with natural language (NLP) features that represent past object state changes and actions. Leveraging the extensive and challenging Ego4D dataset which provides a large-scale collection of first-person perspective videos across numerous interaction scenarios, we introduce an extension noted Ego4D-OSCA that provides new curated annotation data for the object state change anticipation task (OSCA). An extensive experimental evaluation is presented demonstrating the proposed method's efficacy in predicting object state changes in dynamic scenarios. The performance of the proposed approach also underscores the potential of integrating video and linguistic cues to enhance the predictive performance of video understanding systems and lays the groundwork for future research on the new task of object state change anticipation. The source code and the new annotation data (Ego4D-OSCA) will be made publicly available.

Figures (14)

• Figure 1: Examples of modifying actions from the "deform" and "remove" object state change classes represented by a triplet of frames (pre-state, PNR, post-state). Each state change is associated with various actions occurring in diverse environments/scenarios, emphasizing the complexity and challenges introduced in the OSCA problem.
• Figure 2: The intricate relation between verb/object/action and object state change. From left to right: one verb may signify different state changes; different verbs might signify the same state change; an action might lead to a variety of object state changes.
• Figure 3: Two of the nine object state change super-annotated classes in the Ego4D-OSCA dataset, 'deposit' and 'remove'. Pre-/post-state labels for these actions are shown as distinct video segments. 'Deposit' and 'remove' are inverse changes, where pre-deposit matches post-remove, and pre-remove matches post-deposit, indicated by frames and shapes of the same color.
• Figure 4: Annotation pipeline: Occlusions are checked in the pre- & post-frames. A threshold value for the BBOX area of $N$ square pixels (N=100) for each object annotation. Ego4D-SCOD benchmark data are used to automatically annotate the change states per clip.
• Figure 5: Overview of the proposed baseline framework for the object state change anticipation task. The proposed framework anticipates object state changes by integrating real-time visual data and a historical record of past actions and object state changes.