InterPrior: Scaling Generative Control for Physics-Based Human-Object Interactions

TL;DR

InterPrior tackles the challenge of scaling physics-based human-object interaction by learning a unified, generative controller through a three-stage pipeline: 1) distilling a large-scale imitation expert into a goal-conditioned latent policy, 2) applying variational distillation to create a multi-modal, robust motion prior, and 3) RL finetuning to broaden robustness and recover from failures. The approach preserves natural whole-body coordination while expanding task and object coverage, enabling interactions with unseen objects and long-horizon goals. Quantitative and qualitative results show improved success rates, stability under perturbations, and strong generalization, including sim-to-sim transfer and zero-shot adaptation to novel objects. The framework supports interactive steering and offers a reusable prior for humanoid loco-manipulation with potential for sim-to-real deployment and broader HOI applications.

Abstract

Humans rarely plan whole-body interactions with objects at the level of explicit whole-body movements. High-level intentions, such as affordance, define the goal, while coordinated balance, contact, and manipulation can emerge naturally from underlying physical and motor priors. Scaling such priors is key to enabling humanoids to compose and generalize loco-manipulation skills across diverse contexts while maintaining physically coherent whole-body coordination. To this end, we introduce InterPrior, a scalable framework that learns a unified generative controller through large-scale imitation pretraining and post-training by reinforcement learning. InterPrior first distills a full-reference imitation expert into a versatile, goal-conditioned variational policy that reconstructs motion from multimodal observations and high-level intent. While the distilled policy reconstructs training behaviors, it does not generalize reliably due to the vast configuration space of large-scale human-object interactions. To address this, we apply data augmentation with physical perturbations, and then perform reinforcement learning finetuning to improve competence on unseen goals and initializations. Together, these steps consolidate the reconstructed latent skills into a valid manifold, yielding a motion prior that generalizes beyond the training data, e.g., it can incorporate new behaviors such as interactions with unseen objects. We further demonstrate its effectiveness for user-interactive control and its potential for real robot deployment.

Figures (10)

• Figure 1: InterPrior is a versatile generative controller instantiated as a goal-conditioned policy that controls a simulated humanoid to follow goal guidance and interact with objects in a physics-based simulator. Three core, composable capabilities enable pursuing (I) long-horizon snapshot goals, (II) trajectory goals, and (III) contact goals (Top). Yellow, blue, and red dots respectively denote human, object, and contact goals. It demonstrates failure recovery (Bottom Left) from unsuccessful grasps. InterPrior enables steering control from a human operator and can be applied to humanoid robot embodiments (Bottom Right). More demo videos are provided in the https://sirui-xu.github.io/InterPrior.
• Figure 2: Overview of the proposed InterPrior framework. It consists of: (I) full-reference imitation expert training on large-scale human-object interaction data; (II) distillation of the expert into a variational policy with a structured latent space for skill embeddings; and (III) post-training of the variational policy to enhance generalization. Blue modules denote the final policy used at inference; green and red modules are training‑only components, and red arrows denote supervision signals (rewards/losses).
• Figure 3: Qualitative comparison of same reference imitation between InterMimic xu2025intermimic (top) and our InterMimic+ (bottom). InterMimic strictly follows the reference humanoid motion but fails to grasp the thin cloth stand when initialized with perturbations.
• Figure 4: Qualitative results on a multi-object task. The model input is shifted to the second object once the first object is released.
• Figure 5: Zero-shot qualitative results. A single InterPrior model trained from OMOMO li2023object demonstrates generalization to unseen objects and interactions from BEHAVE bhatnagar22behave and HODome zhang2023neuraldome.