Spatial and Surface Correspondence Field for Interaction Transfer

TL;DR

This work tackles interaction transfer between deformable agents and 3D objects by introducing a joint spatial and surface framework built on neural implicit fields. It defines Spatial and Surface Interaction Representation (SSIR) and a Spatial and Surface Correspondence Field (SSCF), leveraging a rotated and deformed implicit field (RDIF) to produce rotation-aware mappings between source and target shapes. A constrained optimization (SSCO) with spatial, surface, and penetration losses, plus a sequence-smooth regularization, enables plausible transfers across challenging geometry and topology changes. Experiments on human-chair and hand-mug tasks show superior performance over surface-only or purely descriptor-based baselines, and ablations confirm the necessity of jointly modeling spatial and surface cues for accurate, realistic interaction transfers, including real-scanned objects.

Abstract

In this paper, we introduce a new method for the task of interaction transfer. Given an example interaction between a source object and an agent, our method can automatically infer both surface and spatial relationships for the agent and target objects within the same category, yielding more accurate and valid transfers. Specifically, our method characterizes the example interaction using a combined spatial and surface representation. We correspond the agent points and object points related to the representation to the target object space using a learned spatial and surface correspondence field, which represents objects as deformed and rotated signed distance fields. With the corresponded points, an optimization is performed under the constraints of our spatial and surface interaction representation and additional regularization. Experiments conducted on human-chair and hand-mug interaction transfer tasks show that our approach can handle larger geometry and topology variations between source and target shapes, significantly outperforming state-of-the-art methods.

Figures (17)

• Figure 1: Method overview. (a) Given a source interaction, our method characterizes it with a combination of spatial and surface representations using the agent points and part of the object points. (b) To transfer the interaction to a target object, we map the agent points and object points related to our interaction representation to the target object space using a learned spatial and surface correspondence field. (c) With the corresponded agent points and object points, we perform an optimization constrained by the spatial and surface representation of the interaction in order to transfer the agent to the target object.
• Figure 2: Object representation. Our method represents the object signed distance field (SDF) as a rotated and deformed neural implicit field. Given an object input, our encoder $E$ encodes it into a rotation-invariant shape code $\bar{\alpha}$ and predicts a rotation matrix $R$, which rotates points $p$ in the object field to canonical posed points $\bar{p}$. Conditioned on the shape code, our decoder $D$ corresponds $\bar{p}$ to the template field $\tilde{p}$ by estimating the deformation $v$ and outputs an SDF correction $\Delta s$. The SDF of $p$ is defined as the SDF of $\tilde{p}$ in the template field $T(\tilde{p})$ plus the correction value $\Delta s$.
• Figure 3: Static interaction transfer results. Our method allows the transfer of interactions to objects with various structures, geometries, and arbitrary orientations.
• Figure 4: More results of our method on static interaction transfer.
• Figure 5: Interaction transfer with real scan inputs. Our method is capable of transferring interactions to scanned objects, enabling varied and realistic interaction transfer in real-world scenarios.