Learning to Visually Connect Actions and their Effects

TL;DR

This work proposes CATE, a framework for visually linking actions to their effects in videos, by formulating Action Selection (AS) and Effect-Affinity Assessment (EAA). It introduces a large cross-sample dataset and a spectrum of baselines, demonstrating that humans outperform models and that analogical reasoning yields the strongest AS performance among baselines. The paper also shows that CATE can serve as a self-supervised pretext task, yielding transferable video representations that improve downstream tasks such as action quality assessment (AQA). Overall, CATE reveals core cognitive mechanisms underlying action-effect understanding and invites future models to leverage these cues for planning, learning from demonstration, and interactive AI systems.

Abstract

We introduce the novel concept of visually Connecting Actions and Their Effects (CATE) in video understanding. CATE can have applications in areas like task planning and learning from demonstration. We identify and explore two different aspects of the concept of CATE: Action Selection (AS) and Effect-Affinity Assessment (EAA), where video understanding models connect actions and effects at semantic and fine-grained levels, respectively. We design various baseline models for AS and EAA. Despite the intuitive nature of the task, we observe that models struggle, and humans outperform them by a large margin. Our experiments show that in solving AS and EAA, models learn intuitive properties like object tracking and pose encoding without explicit supervision. We demonstrate that CATE can be an effective self-supervised task for learning video representations from unlabeled videos. The study aims to showcase the fundamental nature and versatility of CATE, with the hope of inspiring advanced formulations and models.

Figures (13)

• Figure 1: Ability to select the action to carry out to achieve the desired results is so effectively and widely used by humans, that it has become second nature to us---we use it without even realizing we are using. But can the video understanding models do the same thing? Correct answer is (c), not (b).
• Figure 2: Cross-sample Analogical reasoning model. Here, we have shown only one incorrect action; in practice, we use all the incorrect options for counterfactual reasoning.
• Figure 3: Self-supervised pretext task based on connecting actions and effects.Please zoom-in to view better. Applying action can bring the scene from its initial state to the final state, while applying a temporally reversed version of it cannot. State-encoders are frozen; state-encoders and discriminator can be discarded after training---only the action encoders are retained for further usage.
• Figure 4: Self-supervised Effect-Affinity Assessment.Zoom-in to view better. MDR: more directly related effect; LDR: less directly related effect. Degree of how directly related an effect is given on a scale of 1.0 to 0.0 (written above Effect-frames). Farther the effect-frame from the action, lesser directly related it is to the action. Here we have shown cropped video frames to focus on the divers' pose; in practice, we use the entire frame, & the network learns to focus on the diver through Effect-Affinity Assessment-based self-supervision.
• Figure 5: Probing where the model pays attention when connecting action & effects.. (a) Action is 'putting something close to something'. Notice how the model tracks the box moved by the person to follow the state change. Note that the states and the action are from different samples. (b) Where the AR model attends vs where Naive model attends. AR model focuses on state changes and driving action and effects; notice the avocado and coin being tracked. Naive model without reasoning module seems to be doing simple matching based on texture. Zoom in to view better.