ReCorD: Reasoning and Correcting Diffusion for HOI Generation

TL;DR

ReCorD tackles the challenge of accurately rendering human-object interactions in text-to-image diffusion without additional training. By combining Latent Diffusion Models with Visual Language Models and introducing three dedicated modules—Coarse Candidates Generation, Interaction-Aware Reasoning, and Interaction Correcting—ReCorD reasons about pose and layout and refines object placement while preserving pose. Empirical results across HICO-DET, VCOCO, and T2I-CompBench show that ReCorD achieves superior HOI fidelity, especially in Verb CLIP-Score and HOI classification, with competitive image quality metrics and reasonable generation speed, all in a training-free setup. The approach offers a practical, scalable path to accurate HOI synthesis and holds promise for broader interactive scene generation tasks.

Abstract

Diffusion models revolutionize image generation by leveraging natural language to guide the creation of multimedia content. Despite significant advancements in such generative models, challenges persist in depicting detailed human-object interactions, especially regarding pose and object placement accuracy. We introduce a training-free method named Reasoning and Correcting Diffusion (ReCorD) to address these challenges. Our model couples Latent Diffusion Models with Visual Language Models to refine the generation process, ensuring precise depictions of HOIs. We propose an interaction-aware reasoning module to improve the interpretation of the interaction, along with an interaction correcting module to refine the output image for more precise HOI generation delicately. Through a meticulous process of pose selection and object positioning, ReCorD achieves superior fidelity in generated images while efficiently reducing computational requirements. We conduct comprehensive experiments on three benchmarks to demonstrate the significant progress in solving text-to-image generation tasks, showcasing ReCorD's ability to render complex interactions accurately by outperforming existing methods in HOI classification score, as well as FID and Verb CLIP-Score. Project website is available at https://alberthkyhky.github.io/ReCorD/ .

Figures (6)

• Figure 1: Overall architecture of ReCorD. Given a text prompt, ReCorD is structured by three components during the inference of LDM and VLMs, where we first leverage Coarse Candidates Generation $\mathcal{M}_{g}$ to produce coarse candidates. Then, Interaction-aware Reasoning $\mathcal{M}_{r}$ determines the optimal pose and layout regarding to the input. Finally, Interaction Correcting $\mathcal{M}_{c}$ adjusts object placements and maintains the chosen poses to enhance the preliminary images within one generation cycle.
• Figure 2: For a text prompt involving HOI, $\mathcal{M}_g$ attempts to generate coarse candidates by substituting the attention maps obtained from the full prompt (top row) with those derived from the intransitive prompt (bottom row).
• Figure 3: Given $k$ coarse images, the Pose Selection Agent identifies the image most closely aligning with the text prompt $y$, and the Layout Agent updates the object's position $\hat{b}_o$ by reasoning arrangements while preserving the pose $\mathcal{P}$.
• Figure 4: $\mathcal{M}_c$ refine the coarse candidate according to the suggested layout by adjusting object location and size based on Eq. \ref{['eq:box_constraints']}, employing the inverse mask $\bar{A}_m$ along with an element-wise product to deal with the attention overlapping concerns.
• Figure 5: Visual comparison with existing baselines for HICO-DET (a-c) and VCOCO (d-f) using different text prompts, where ReCorD attains better delineation of interaction, and renders images matching the text instructions. (a) a young man is signinga sports ball. (b) a woman is carryinga pizza. (c) a boy is chasinga bird. (d) a child is cuttinga cake. (e) a toddler is pointing ata laptop. (f) a woman is holdinga fork. The bounding boxes on the results of L2I models are additional input for HOI generation.