Re-HOLD: Video Hand Object Interaction Reenactment via adaptive Layout-instructed Diffusion Model

TL;DR

HOI video generation remains challenging due to occlusions and variability in object shape and size. The authors propose Re-HOLD, a two-branch diffusion framework that uses specialized hand and object layouts to disentangle HOI signals, augmented by a Hand-Object Interaction Restoration module with separate memory banks and Attention, and an adaptive layout adjustment strategy for cross-object reenactment. They show state-of-the-art performance on self-reenactment and cross-object reenactment across HOI4D-derived data, with superior hand fidelity, object texture, and temporal coherence. This work advances practical, controllable HOI video synthesis for digital humans, though limitations include 3D object manipulation and broader societal considerations.

Abstract

Current digital human studies focusing on lip-syncing and body movement are no longer sufficient to meet the growing industrial demand, while human video generation techniques that support interacting with real-world environments (e.g., objects) have not been well investigated. Despite human hand synthesis already being an intricate problem, generating objects in contact with hands and their interactions presents an even more challenging task, especially when the objects exhibit obvious variations in size and shape. To tackle these issues, we present a novel video Reenactment framework focusing on Human-Object Interaction (HOI) via an adaptive Layout-instructed Diffusion model (Re-HOLD). Our key insight is to employ specialized layout representation for hands and objects, respectively. Such representations enable effective disentanglement of hand modeling and object adaptation to diverse motion sequences. To further improve the generation quality of HOI, we design an interactive textural enhancement module for both hands and objects by introducing two independent memory banks. We also propose a layout adjustment strategy for the cross-object reenactment scenario to adaptively adjust unreasonable layouts caused by diverse object sizes during inference. Comprehensive qualitative and quantitative evaluations demonstrate that our proposed framework significantly outperforms existing methods. Project page: https://fyycs.github.io/Re-HOLD.

Figures (5)

• Figure 1: Cross-Reenactment Results by our Re-HOLD framework. Given a driving video and a target object, Re-HOLD can synthesize high-fidelity Human-Object Interaction (HOI) videos, even when the sizes of the target and original object differ significantly.
• Figure 2: Overview of our proposed Re-HOLD framework. We propose a two-branch framework that consists of a Reference U-Net and a Denoising U-Net. The Reference U-Net takes a reference object image for object texture encoding while the denoising one takes noise latent and layout guidance as input for diffusion processing. To enhance the quality of HOI generation, we adopt the HOI Restoration Module for hand information and fine-grained object information restoration.
• Figure 3: Schematic diagram of adaptive layout adjustment.
• Figure 4: Qualitative results compared with other methods. Our approach achieves high-fidelity HOI details and satisfactory image quality in both self-reenactment and cross-object reenactment settings.
• Figure 5: Qualitative ablation results of cross-reenactment when removing different components in our framework.