Diff-IP2D: Diffusion-Based Hand-Object Interaction Prediction on Egocentric Videos

TL;DR

Diff-IP2D introduces a diffusion-based, iterative non-autoregressive framework for jointly predicting future hand trajectories and object affordances from 2D egocentric videos. It projects input frames into latent HOI features, denoises them with a Motion-Aware Denoising Transformer guided by egomotion-derived homography features, and decodes predictions via dedicated heads for hand motion and object contact points. The approach leverages bidirectional latent constraints and dense latent supervision to mitigate error accumulation and better capture high-level human intent, outperforming autoregressive baselines on EK55, EK100, and Ego datasets with strong generalization. The method offers practical impact for service robotics and extended reality, enabling more reliable anticipation of hand-object interactions in visually dynamic, first-person settings.

Abstract

Understanding how humans would behave during hand-object interaction is vital for applications in service robot manipulation and extended reality. To achieve this, some recent works have been proposed to simultaneously forecast hand trajectories and object affordances on human egocentric videos. The joint prediction serves as a comprehensive representation of future hand-object interactions in 2D space, indicating potential human motion and motivation. However, the existing approaches mostly adopt the autoregressive paradigm for unidirectional prediction, which lacks mutual constraints within the holistic future sequence, and accumulates errors along the time axis. Meanwhile, these works basically overlook the effect of camera egomotion on first-person view predictions. To address these limitations, we propose a novel diffusion-based interaction prediction method, namely Diff-IP2D, to forecast future hand trajectories and object affordances concurrently in an iterative non-autoregressive manner. We transform the sequential 2D images into latent feature space and design a denoising diffusion model to predict future latent interaction features conditioned on past ones. Motion features are further integrated into the conditional denoising process to enable Diff-IP2D aware of the camera wearer's dynamics for more accurate interaction prediction. Extensive experiments demonstrate that our method significantly outperforms the state-of-the-art baselines on both the off-the-shelf metrics and our newly proposed evaluation protocol. This highlights the efficacy of leveraging a generative paradigm for 2D hand-object interaction prediction. The code of Diff-IP2D is released as open source at https://github.com/IRMVLab/Diff-IP2D.

Figures (12)

• Figure 1: Autoregressive generation vs. parallel generation.
• Figure 2: An example to clarify our motivation to propose an iter-NAR paradigm considering bidirectional constraints. The hand waypoints from ground-truth labels and HOI prediction approaches are connected by blue and white dashed lines respectively. Note that we reverse the RGB values of each image to display the arm’s positions more clearly. There is a lack of backward constraints in AR-based USST bao2023uncertainty, leading to a shorter predicted trajectory (almost curled up into a point) and larger accumulated displacement errors. In contrast, our Diff-IP2D with iter-NAR paradigm is potentially guided by final HOI states, and thus predicts more accurate hand trajectories following both spatial causality and temporal causality.
• Figure 3: Ablation study on observation time. The reference line represents the performance of the second-best baseline trained from scratch using $10$ observation frames.
• Figure 4: Visualization of object affordance prediction grounded on frames of Epic-Kitchens. The ground-truth contact points are represented by red dots. The contact points predicted by our Diff-IP2D with average distances to the ground-truth points are represented by white dots. The counterparts predicted by OCT liu2022joint are represented by blue dots.
• Figure 5: Two additional explanatory cases.