Mask2IV: Interaction-Centric Video Generation via Mask Trajectories

TL;DR

Mask2IV tackles interaction-centric video generation by decoupling trajectory modeling from video synthesis. It first predicts mask-based interaction trajectories for both actors and objects, conditioned on either a text description or a target object position, then generates the final video guided by these trajectories. The approach is validated on two benchmarks—HOI4D for human-object interactions and BridgeDataV2 for robotic manipulation—showing superior realism and controllability compared with baselines. This framework provides a practical, user-friendly way to synthesize diverse, physically plausible interaction videos for embodied AI tasks such as imitation learning and affordance reasoning.

Abstract

Generating interaction-centric videos, such as those depicting humans or robots interacting with objects, is crucial for embodied intelligence, as they provide rich and diverse visual priors for robot learning, manipulation policy training, and affordance reasoning. However, existing methods often struggle to model such complex and dynamic interactions. While recent studies show that masks can serve as effective control signals and enhance generation quality, obtaining dense and precise mask annotations remains a major challenge for real-world use. To overcome this limitation, we introduce Mask2IV, a novel framework specifically designed for interaction-centric video generation. It adopts a decoupled two-stage pipeline that first predicts plausible motion trajectories for both actor and object, then generates a video conditioned on these trajectories. This design eliminates the need for dense mask inputs from users while preserving the flexibility to manipulate the interaction process. Furthermore, Mask2IV supports versatile and intuitive control, allowing users to specify the target object of interaction and guide the motion trajectory through action descriptions or spatial position cues. To support systematic training and evaluation, we curate two benchmarks covering diverse action and object categories across both human-object interaction and robotic manipulation scenarios. Extensive experiments demonstrate that our method achieves superior visual realism and controllability compared to existing baselines.

Figures (6)

• Figure 1: Comparison on control signal acquisition. Hand-mask-controlled video generation methods require users to provide dense hand mask sequences as input. In contrast, Mask2IV autonomously generates trajectories for both hands and objects without manual annotation, and can adaptively produce different trajectories based on the specified object.
• Figure 2: Mask2IV synthesizes videos of human hands or robot arms interacting with a specified object, indicated by an input mask. It first predicts a mask-based interaction trajectory (visualized in the top-left corner of each frame), and then generates the video guided by this trajectory. The generation process is conditioned on either a text prompt or a target position mask.
• Figure 3: The framework of Mask2IV. It consists of two stages: Interaction Trajectory Generation and Trajectory-conditioned Video Generation. The first stage produces a mask-based interaction trajectory, while the second stage synthesizes a video conditioned on the predicted trajectory.
• Figure 4: Qualitative comparison with CosHand and InterDyn. Generation artifacts are marked with yellow circles.
• Figure 5: Qualitative analysis of interaction trajectory generation involving different objects. Target objects are highlighted in red in the initial image.