Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

InstructMoLE: Instruction-Guided Mixture of Low-rank Experts for Multi-Conditional Image Generation

Jinqi Xiao, Qing Yan, Liming Jiang, Zichuan Liu, Hao Kang, Shen Sang, Tiancheng Zhi, Jing Liu, Cheng Yang, Xin Lu, Bo Yuan

TL;DR

InstructMoLE addresses task interference in parameter-efficient diffusion fine-tuning by replacing token-level routing with a global, instruction-guided routing policy (IGR) and by enforcing functional diversity with an output-space orthogonality loss. A Perceiver-based distillation pipeline fuses token-level instruction features and holistic CLIP semantics to produce a robust global routing signal Z_global, which selects a single expert council per layer broadcast across all tokens. The approach yields state-of-the-art performance on multi-conditional image generation benchmarks (OmniContext, XVerseBench, GEdit-EN-full) while maintaining efficiency, reducing memory usage, and preserving global instruction fidelity. These results demonstrate the importance of global, instruction-aware routing for complex compositional control in diffusion models and offer a generalizable framework for instruction-driven fine-tuning in vision-language tasks.

Abstract

Parameter-Efficient Fine-Tuning of Diffusion Transformers (DiTs) for diverse, multi-conditional tasks often suffers from task interference when using monolithic adapters like LoRA. The Mixture of Low-rank Experts (MoLE) architecture offers a modular solution, but its potential is usually limited by routing policies that operate at a token level. Such local routing can conflict with the global nature of user instructions, leading to artifacts like spatial fragmentation and semantic drift in complex image generation tasks. To address these limitations, we introduce InstructMoLE, a novel framework that employs an Instruction-Guided Mixture of Low-Rank Experts. Instead of per-token routing, InstructMoLE utilizes a global routing signal, Instruction-Guided Routing (IGR), derived from the user's comprehensive instruction. This ensures that a single, coherently chosen expert council is applied uniformly across all input tokens, preserving the global semantics and structural integrity of the generation process. To complement this, we introduce an output-space orthogonality loss, which promotes expert functional diversity and mitigates representational collapse. Extensive experiments demonstrate that InstructMoLE significantly outperforms existing LoRA adapters and MoLE variants across challenging multi-conditional generation benchmarks. Our work presents a robust and generalizable framework for instruction-driven fine-tuning of generative models, enabling superior compositional control and fidelity to user intent.

InstructMoLE: Instruction-Guided Mixture of Low-rank Experts for Multi-Conditional Image Generation

TL;DR

InstructMoLE addresses task interference in parameter-efficient diffusion fine-tuning by replacing token-level routing with a global, instruction-guided routing policy (IGR) and by enforcing functional diversity with an output-space orthogonality loss. A Perceiver-based distillation pipeline fuses token-level instruction features and holistic CLIP semantics to produce a robust global routing signal Z_global, which selects a single expert council per layer broadcast across all tokens. The approach yields state-of-the-art performance on multi-conditional image generation benchmarks (OmniContext, XVerseBench, GEdit-EN-full) while maintaining efficiency, reducing memory usage, and preserving global instruction fidelity. These results demonstrate the importance of global, instruction-aware routing for complex compositional control in diffusion models and offer a generalizable framework for instruction-driven fine-tuning in vision-language tasks.

Abstract

Parameter-Efficient Fine-Tuning of Diffusion Transformers (DiTs) for diverse, multi-conditional tasks often suffers from task interference when using monolithic adapters like LoRA. The Mixture of Low-rank Experts (MoLE) architecture offers a modular solution, but its potential is usually limited by routing policies that operate at a token level. Such local routing can conflict with the global nature of user instructions, leading to artifacts like spatial fragmentation and semantic drift in complex image generation tasks. To address these limitations, we introduce InstructMoLE, a novel framework that employs an Instruction-Guided Mixture of Low-Rank Experts. Instead of per-token routing, InstructMoLE utilizes a global routing signal, Instruction-Guided Routing (IGR), derived from the user's comprehensive instruction. This ensures that a single, coherently chosen expert council is applied uniformly across all input tokens, preserving the global semantics and structural integrity of the generation process. To complement this, we introduce an output-space orthogonality loss, which promotes expert functional diversity and mitigates representational collapse. Extensive experiments demonstrate that InstructMoLE significantly outperforms existing LoRA adapters and MoLE variants across challenging multi-conditional generation benchmarks. Our work presents a robust and generalizable framework for instruction-driven fine-tuning of generative models, enabling superior compositional control and fidelity to user intent.
Paper Structure (44 sections, 2 theorems, 8 equations, 12 figures, 6 tables)

This paper contains 44 sections, 2 theorems, 8 equations, 12 figures, 6 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Conditional Generation with Diffusion Transformers.
  4. Mixture-of-Experts for Parameter-Efficient Fine-Tuning.
  5. MoE Design: Routing Policies and Expert Diversity.
  6. Methodology
  7. InstructMoLE Architecture
  8. Token-level vs. Instance-level Routing.
  9. Instruction-Guided Routing (IGR) Policy
  10. Global Routing Signal ($\mathbf{Z}_{\text{global}}$).
  11. IGR Gating Mechanism.
  12. Theoretical Analysis
  13. Training Objectives
  14. Generation Accuracy ($\mathcal{L}_{\text{flow}}$).
  15. Expert Load Balancing ($\mathcal{L}_{\text{aux}}$).
  16. ...and 29 more sections

Key Result

Proposition 1

The orthogonality loss $\mathcal{L}_{\text{ortho}}$ enforces a full-rank functional basis over the expert outputs, satisfying the expert diversification requirement established in li2025theory.

Figures (12)

  • Figure 1: Illustration of the InstructMoLE framework. A global signal, ${\mathbf{Z}_\text{global}}$, is distilled from the user's instruction to guide a Router. The Router selects a single, consistent set of LoRA experts, which is then applied to all input tokens.
  • Figure 2: Qualitative comparison with state-of-the-art models.
  • Figure 3: Qualitative comparison of in-context generation on OmniContext benchmark.
  • Figure 4: Qualitative comparison of multi-subject driven generation on XVerse benchmark.
  • Figure 5: Qualitative comparison of single-image editing on GEdit-EN-full benchmark.
  • ...and 7 more figures

Theorems & Definitions (4)

  • Proposition 1: Expert Diversity
  • Proof 1
  • Proposition 2: Spatial Consistency
  • Proof 2