Preventing Shortcuts in Adapter Training via Providing the Shortcuts

TL;DR

This work addresses the entanglement problem in adapter-based T2I personalization, where single-image reconstruction causes the adapter to encode both the target factor $T$ and confounds $C$ such as pose, expression, and distribution. It introduces Shortcut-Rerouted (SR) adapter training, which routes confounds through auxiliary modules $\mathcal{S}_C$ (e.g., LoRA or ControlNet) during training so the main adapter $\mathcal{A}$ focuses on $T$; at inference, the SR module is removed to restore the original generator with disentangled identity control. Two instantiations are explored: SR-LoRA for distribution shifts and SR-CN for pose/expression leakage, implemented in face and full-body personalization scenarios. Across quantitative metrics and qualitative results, SR adapters achieve improved prompt adherence, stronger prior preservation, and enhanced generation quality and diversity, illustrating a general modular principle for disentangled representations in large models.

Abstract

Adapter-based training has emerged as a key mechanism for extending the capabilities of powerful foundation image generators, enabling personalized and stylized text-to-image synthesis. These adapters are typically trained to capture a specific target attribute, such as subject identity, using single-image reconstruction objectives. However, because the input image inevitably contains a mixture of visual factors, adapters are prone to entangle the target attribute with incidental ones, such as pose, expression, and lighting. This spurious correlation problem limits generalization and obstructs the model's ability to adhere to the input text prompt. In this work, we uncover a simple yet effective solution: provide the very shortcuts we wish to eliminate during adapter training. In Shortcut-Rerouted Adapter Training, confounding factors are routed through auxiliary modules, such as ControlNet or LoRA, eliminating the incentive for the adapter to internalize them. The auxiliary modules are then removed during inference. When applied to tasks like facial and full-body identity injection, our approach improves generation quality, diversity, and prompt adherence. These results point to a general design principle in the era of large models: when seeking disentangled representations, the most effective path may be to establish shortcuts for what should NOT be learned.

Figures (11)

• Figure 1: Shortcut Rerouting re-enables text control of pose and expression after adapter training. In the context of personalized generation, without shortcut rerouting, the adapter overfits to the reference image and reproduces its pose and expression, ignoring the prompt. With Shortcut-Rerouted Adapter Training, the adapter disentangles identity from other factors, allowing the model to respond faithfully to prompt-specified expressions and head poses. This restores compositionality, preserves the prior, and leads to more expressive and diverse generations.
• Figure 2: Common adapter training is susceptible to learning undesired shortcutes. The common single-image reconstruction objective used in adapter training inadvertently encourages the adapter to pick up all the attributes in the adapter input (e.g. pose, expression, background, distribution) and leak them into the generation. While some confounding attributes like background can be factored out using masking, many other cannot. This makes learning a pure "identity" adapter challenging.
• Figure 3: Method. The Shortcut-Rerouting (SR)-Module serves as a generic shortcut adapter that can take various forms—such as a ControlNet, LoRA, or IP-Adapter—depending on the confounding factor being addressed (e.g., pose, distribution, or style). Illustrated here is the case of SR with ControlNet, where pose and expression cues are explicitly rerouted via the ControlNet during adapter training. At inference time, the ControlNet is removed, restoring independent pose and expression control from the text prompt alone.
• Figure 4: Distribution shift between model and finetuning dataset. Due to the distribution shift, directly training a personalization adapter on the finetuning dataset leads to degraded quality.
• Figure 5: Qualitative comparison of different "face" adapters.Top: close-up portraits with varied expressions. Bottom: full-body generations. Our approach preserves the visual prior more faithfully, enabling expressive and identity-consistent personalized image generation.