InteractDiffusion: Interaction Control in Text-to-Image Diffusion Models

TL;DR

InteractDiffusion introduces a pluggable Interaction Module for text-to-image diffusion models to condition generation on Human-Object Interactions (HOIs) specified as triplets with bounding boxes. It tokenizes HOI information (InToken), encodes relational structure via instance and role embeddings (InBedding), and injects interaction cues through an Interaction Self-Attention block (InFormer) into the diffusion UNet, preserving base model knowledge. Evaluated on HICO-DET, it achieves state-of-the-art HOI detection scores while maintaining high image fidelity (FID/KID) and shows good transferability to personalized Stable Diffusion models and reasonable zero-shot generalization. This work broadens T2I controllability to dynamic interactions, enabling more realistic scenes with interacting agents and objects in generated content.

Abstract

Large-scale text-to-image (T2I) diffusion models have showcased incredible capabilities in generating coherent images based on textual descriptions, enabling vast applications in content generation. While recent advancements have introduced control over factors such as object localization, posture, and image contours, a crucial gap remains in our ability to control the interactions between objects in the generated content. Well-controlling interactions in generated images could yield meaningful applications, such as creating realistic scenes with interacting characters. In this work, we study the problems of conditioning T2I diffusion models with Human-Object Interaction (HOI) information, consisting of a triplet label (person, action, object) and corresponding bounding boxes. We propose a pluggable interaction control model, called InteractDiffusion that extends existing pre-trained T2I diffusion models to enable them being better conditioned on interactions. Specifically, we tokenize the HOI information and learn their relationships via interaction embeddings. A conditioning self-attention layer is trained to map HOI tokens to visual tokens, thereby conditioning the visual tokens better in existing T2I diffusion models. Our model attains the ability to control the interaction and location on existing T2I diffusion models, which outperforms existing baselines by a large margin in HOI detection score, as well as fidelity in FID and KID. Project page: https://jiuntian.github.io/interactdiffusion.

Figures (8)

• Figure 1: Comparison between InteractDiffusion and existing baselines in terms of generated image quality scores in FID and KID and HOI detection score in mAP. GLIGEN* is HICO-DET fine-tuned GLIGEN model. The last row shows the Detection Score from real images.
• Figure 2: The overall framework of InteractDiffusion. Our proposed pluggable Interaction Module $I$ seamlessly incorporates interaction information into an existing T2I diffusion model (left). The proposed module $I$ (right) consists of Interaction Tokenizer (\ref{['subsec:interaction_tokenizer']}) that transforms interaction information into meaningful tokens, Interaction Embedding (\ref{['subsec:interaction_embed']}) that incorporates intricate interaction relationship, and Interaction Self-Attention (\ref{['subsec:interaction_transformer']}) that integrates interaction control information into Visual Tokens of the existing T2I diffusion model.
• Figure 3: "Between" operation obtains the action focus area (highlighted in orange) between subject and object bounding boxes.
• Figure 4: Interaction Tokenizer. View bottom-up.
• Figure 5: Interaction Embeddings. Learnable instance embedding $q$ and role embedding $r$ are added to tokens to represent intricate interaction relationships between subject $s$, action $a$ and object $o$.