GeneOH Diffusion: Towards Generalizable Hand-Object Interaction Denoising via Denoising Diffusion

TL;DR

This work tackles the challenging task of denoising hand-object interaction sequences by introducing GeneOH Diffusion, a framework that unites a Generalized contact-centric HOI representation (GeneOH) with a domain-generalizable denoising scheme based on diffusion. The GeneOH representation encodes hand trajectories and spatial/temporal HOI relations with canonicalization to improve cross-interaction generalization. A canonical denoiser operating in a whitened noise space, together with a progressive three-stage denoising pipeline (MotionDiff, SpatialDiff, TemporalDiff), enables robust denoising across unseen objects and varied noise patterns. Empirical results on GRAB, GRAB Beta, HOI4D, and ARCTIC demonstrate superior denoising performance and versatility, including stochastic multi-solution denoising and downstream applications such as refining noisy hand trajectories and aiding grasp/manipulation synthesis. The approach promises practical impact for robust HOI tracking and data-driven downstream tasks, while noting limitations around accurately modeling object poses in-the-wild scenarios.”

Abstract

In this work, we tackle the challenging problem of denoising hand-object interactions (HOI). Given an erroneous interaction sequence, the objective is to refine the incorrect hand trajectory to remove interaction artifacts for a perceptually realistic sequence. This challenge involves intricate interaction noise, including unnatural hand poses and incorrect hand-object relations, alongside the necessity for robust generalization to new interactions and diverse noise patterns. We tackle those challenges through a novel approach, GeneOH Diffusion, incorporating two key designs: an innovative contact-centric HOI representation named GeneOH and a new domain-generalizable denoising scheme. The contact-centric representation GeneOH informatively parameterizes the HOI process, facilitating enhanced generalization across various HOI scenarios. The new denoising scheme consists of a canonical denoising model trained to project noisy data samples from a whitened noise space to a clean data manifold and a "denoising via diffusion" strategy which can handle input trajectories with various noise patterns by first diffusing them to align with the whitened noise space and cleaning via the canonical denoiser. Extensive experiments on four benchmarks with significant domain variations demonstrate the superior effectiveness of our method. GeneOH Diffusion also shows promise for various downstream applications. Project website: https://meowuu7.github.io/GeneOH-Diffusion/.

Figures (22)

• Figure 1: Trained only on limited data, https://meowuu7.github.io/GeneOH-Diffusion/ can clean novel noisy interactions with new objects, hand motions, and unseen noise patterns (Fig. (a)), produces diverse refined trajectories with discrete manipulation modes (Fig. (b)), and is a practical tool for many applications (Fig. (c)).
• Figure 2: Three components of GeneOH.
• Figure 3: The progressive HOI denoosing gradually cleans the input noisy trajectory through three stages. Each stage concentrates on refining the trajectory by denoising a specific part of GeneOH via a canonical denoiser through the "denoising via diffusion" strategy.
• Figure 4: Qualitative comparisons. Please refer to https://meowuu7.github.io/GeneOH-Diffusion/ and https://youtu.be/ySwkFPJVhHYfor animated results.
• Figure 5: Stochastic denoising can produce divserse results with discrete modes.

Theorems & Definitions (2)

• Claim 1
• Claim 2