Compass Control: Multi Object Orientation Control for Text-to-Image Generation

TL;DR

Compass Control tackles the lack of explicit per-object 3D orientation control in text-to-image diffusion. It introduces per-object compass tokens $c_n$ derived from orientation angles $\theta_n$, and uses Coupled Attention Localization (CALL) to constrain cross-attention within object regions, enabling disentangled control across multiple objects. The method is trained on a synthetic dataset with staged training and LoRA fine-tuning, and it supports personalization via a small set of unposed images. Experiments show state-of-the-art orientation control and text alignment, strong generalization to unseen categories, and effective personalization, with user studies confirming perceptual quality. Limitations include occlusion and high overlap scenarios, suggesting future work on more complex 3D conditioning and non-rigid objects.

Abstract

Existing approaches for controlling text-to-image diffusion models, while powerful, do not allow for explicit 3D object-centric control, such as precise control of object orientation. In this work, we address the problem of multi-object orientation control in text-to-image diffusion models. This enables the generation of diverse multi-object scenes with precise orientation control for each object. The key idea is to condition the diffusion model with a set of orientation-aware \textbf{compass} tokens, one for each object, along with text tokens. A light-weight encoder network predicts these compass tokens taking object orientation as the input. The model is trained on a synthetic dataset of procedurally generated scenes, each containing one or two 3D assets on a plain background. However, direct training this framework results in poor orientation control as well as leads to entanglement among objects. To mitigate this, we intervene in the generation process and constrain the cross-attention maps of each compass token to its corresponding object regions. The trained model is able to achieve precise orientation control for a) complex objects not seen during training and b) multi-object scenes with more than two objects, indicating strong generalization capabilities. Further, when combined with personalization methods, our method precisely controls the orientation of the new object in diverse contexts. Our method achieves state-of-the-art orientation control and text alignment, quantified with extensive evaluations and a user study.

Figures (23)

• Figure 1: We present Compass Control, a method to generate multi-object scenes with orientation control from text-to-image diffusion models. Given a text prompt and an orientation of each object (shown as frustum, the colored face is the forward direction), our method generates scenes that align with both the prompt and specified orientations. Additionally, with a few ($\approx10$) unposed images of a new object, our model is personalized to generate the object in target orientations.
• Figure 2: Synthetic data generation. We curate 10 diverse 3D assets, and render them in diverse layouts and orientations in Blender blender. The rendered scenes are augmented with realistic generations from Canny canny1986computational ControlNet controlnet. The final dataset consists of one and two object scenes.
• Figure 3: Compass Control. Given an orientation angle $\theta_j$, we project it to a compass token with a lightweight encoder model. The compass tokens are interleaved with the text tokens (as shown in the figure) and passed through the text encoder. The outputs of the text encoder are used to condition the denoising process in the U-Net. We train $\mathcal{P}$ and also fine-tune the U-Net using LoRA hu2021lora.
• Figure 4: Binding the compass tokens: We visualize the averaged cross attention of the compass token(s) when training with CALL (shown on the left) and without it (shown on the right). CALL localizes the influence of the compass token at the correct regions, which (a) improves orientation control (b) disentangles orientations in multi-object scenes. In (b), $\mathbf{c_1}$ and $\mathbf{c_2}$ are compass tokens for car and motorbike, respectively.
• Figure 5: Staged training results in improved adherence of objects to the bounding boxes, leading to orientation learning.