Break-for-Make: Modular Low-Rank Adaptations for Composable Content-Style Customization

TL;DR

The paper tackles the entanglement of content and style in text-to-image customization by introducing Break-for-Make, a two-stage learning framework that disentangles content and style through Partly Learnable Projection (PLP) and Multi-Correspondence Projection Learning (MCP) within low-rank adapters. In Stage 1, PLP partitions the adaptation parameters into content- and style-specific subspaces with orthogonally initialized, partially frozen blocks to enable clean separation; MCP further guards generalization by diversifying content-style pairings. In Stage 2, the separate PLPs are recombined into a unified fusion adapter and briefly fine-tuned to produce high-fidelity content-style images; the approach supports both joint and individual generation of content or style when needed. Across textures, materials, and artistic styles, Break-for-Make outperforms state-of-the-art two-stage and merging baselines in content-style alignment and fidelity, while offering improved editability and robust disentanglement. These results indicate a practical, modular path toward composable content-style customization in diffusion-based generation with reduced interference between concepts, enabling more reliable designer-driven visual synthesis.

Abstract

Personalized generation paradigms empower designers to customize visual intellectual properties with the help of textual descriptions by tuning or adapting pre-trained text-to-image models on a few images. Recent works explore approaches for concurrently customizing both content and detailed visual style appearance. However, these existing approaches often generate images where the content and style are entangled. In this study, we reconsider the customization of content and style concepts from the perspective of parameter space construction. Unlike existing methods that utilize a shared parameter space for content and style, we propose a learning framework that separates the parameter space to facilitate individual learning of content and style, thereby enabling disentangled content and style. To achieve this goal, we introduce "partly learnable projection" (PLP) matrices to separate the original adapters into divided sub-parameter spaces. We propose "break-for-make" customization learning pipeline based on PLP, which is simple yet effective. We break the original adapters into "up projection" and "down projection", train content and style PLPs individually with the guidance of corresponding textual prompts in the separate adapters, and maintain generalization by employing a multi-correspondence projection learning strategy. Based on the adapters broken apart for separate training content and style, we then make the entity parameter space by reconstructing the content and style PLPs matrices, followed by fine-tuning the combined adapter to generate the target object with the desired appearance. Experiments on various styles, including textures, materials, and artistic style, show that our method outperforms state-of-the-art single/multiple concept learning pipelines in terms of content-style-prompt alignment.

Figures (17)

• Figure 1: Frameworks of the two main approaches and ours for customized content-style image generation. LoRA joint training incorporates image-text pairs to fine-tuning the overall model parameters. ZipLoRA shah2023ziplora effectively merging independently trained content and style LoRAs, then add to per-trained weights to generaet images of the customized content and style. Our method trains content and style in separated parameter subspaces of LoRA, results in disentanglement of content and style while maintaining high level of fidelity.
• Figure 2: Analysis of the two main methods and ours using LoRA for customized content-style image generation. Joint training LoRA will mix the parameter space of content and style, leads to entanglement of content and style. Merging LoRAs after independent training has problem of conflict parameters from content LoRA and style LoRA, leads to content and/or style unfaithful after fusion. Our proposed method trains the content and style in separate parameter subspaces of the LoRA modules, with orthogonal fixed parameter spaces, resulting in disentangled and faithful fusion of content and style.
• Figure 3: Illustration of the multi-correspondence projection. We present the learned content distribution on the left of the top row. When training specific content and style in a one-to-one manner, the content will tend to overfit to the specific style, as illustrated on the middle of the top row. By leveraging our proposed multi-correspondence projection, we learn multiple styles with the content in PLP, enhance the generalization of the learned content.
• Figure 4: Qualitative evaluation and comparison of DB+LoRA, TI, ProSpect, CD, ZipLoRA and our method in diverse styles. We present the results of customized generation of the same content and different styles. Results indicate that our method generates harmonious fusion images of the content and the style, while preserving the disentanglement of content and style, as well as maintaining high-level fidelity of them.
• Figure 5: Qualitative evaluation and comparison of DB+LoRA, TI, ProSpect, CD, ZipLoRA, and our method in diverse contents. The results indicate that our method generates harmonious content-style fusion images with diverse contents while preserving the disentanglement of content and style, as well as maintaining high-level fidelity.