Laytrol: Preserving Pretrained Knowledge in Layout Control for Multimodal Diffusion Transformers

TL;DR

Laytrol tackles the problem of preserving pretrained knowledge in layout-conditioned diffusion transformers by copying parameters from a pretrained MM-DiT into a dedicated layout control network and using a careful initialization strategy. To reduce distribution shift, LaySyn synthesizes training data with the base model and augments prompts with layout cues. A layout-level Rotary Position Embedding and randomized prompt dropping reinforce spatial control while preserving base-model quality. Experimental results on T2I-CompBench and COCO 2017 demonstrate superior layout fidelity and image quality compared to state-of-the-art layout-to-image methods, validating the effectiveness of pretrained-knowledge preservation for controllable generation.

Abstract

With the development of diffusion models, enhancing spatial controllability in text-to-image generation has become a vital challenge. As a representative task for addressing this challenge, layout-to-image generation aims to generate images that are spatially consistent with the given layout condition. Existing layout-to-image methods typically introduce the layout condition by integrating adapter modules into the base generative model. However, the generated images often exhibit low visual quality and stylistic inconsistency with the base model, indicating a loss of pretrained knowledge. To alleviate this issue, we construct the Layout Synthesis (LaySyn) dataset, which leverages images synthesized by the base model itself to mitigate the distribution shift from the pretraining data. Moreover, we propose the Layout Control (Laytrol) Network, in which parameters are inherited from MM-DiT to preserve the pretrained knowledge of the base model. To effectively activate the copied parameters and avoid disturbance from unstable control conditions, we adopt a dedicated initialization scheme for Laytrol. In this scheme, the layout encoder is initialized as a pure text encoder to ensure that its output tokens remain within the data domain of MM-DiT. Meanwhile, the outputs of the layout control network are initialized to zero. In addition, we apply Object-level Rotary Position Embedding to the layout tokens to provide coarse positional information. Qualitative and quantitative experiments demonstrate the effectiveness of our method.

Figures (7)

• Figure 1: Layout-to-image results of Laytrol. The text in the same color as the bounding box corresponds to the local prompt. Laytrol enables layout-conditioned control in MM-DiT while effectively retaining the knowledge learned during the pre-training stage, and simultaneously mitigates the domain shift introduced by fine-tuning.
• Figure 2: Overview of Layout Control (Laytrol) pipeline. (a) Laytrol blocks inherit vision-language pre-trained knowledge from DiT via parameter copying, which facilitates learning layout-conditioned control. Setting the global prompt tokens to null tokens encourages the model to focus more on the layout tokens during training. (b) The layout encoder is initialized as a pure text encoder with a zero-initialized projection layer. (c) The coordinates of the patch containing the bounding box center are used for applying RoPE to the layout tokens.
• Figure 3: The pipeline for constructing the LaySyn dataset.
• Figure 4: Qualitative comparison with other other methods. From the results, Laytrol shows better performance in terms of stylistic consistency, layout realism and object aesthetics.
• Figure 5: An example data instance from LaySyn dataset.