3SGen: Unified Subject, Style, and Structure-Driven Image Generation with Adaptive Task-specific Memory

TL;DR

The paper tackles entanglement among subject, style, and structure conditioning in image generation by proposing 3SGen, a unified framework that uses a multimodal large language model with learnable semantic queries, a VAE branch for fine-grained details, and an Adaptive Task-specific Memory to store and retrieve task-specific priors. It introduces ATM with an adaptive gate to enable dynamic, per-task conditioning and compositional generation, achieving scalable cross-task performance. The authors also present 3SGen-Bench, a unified, multimodal benchmark with standardized metrics for evaluating cross-task fidelity and controllability. Extensive experiments on 3SGen-Bench and public benchmarks demonstrate superior controllability, fidelity, and cross-task generalization across subject-, style-, and structure-driven generation, validating the approach's practical impact for universal image-driven generation.

Abstract

Recent image generation approaches often address subject, style, and structure-driven conditioning in isolation, leading to feature entanglement and limited task transferability. In this paper, we introduce 3SGen, a task-aware unified framework that performs all three conditioning modes within a single model. 3SGen employs an MLLM equipped with learnable semantic queries to align text-image semantics, complemented by a VAE branch that preserves fine-grained visual details. At its core, an Adaptive Task-specific Memory (ATM) module dynamically disentangles, stores, and retrieves condition-specific priors, such as identity for subjects, textures for styles, and spatial layouts for structures, via a lightweight gating mechanism along with several scalable memory items. This design mitigates inter-task interference and naturally scales to compositional inputs. In addition, we propose 3SGen-Bench, a unified image-driven generation benchmark with standardized metrics for evaluating cross-task fidelity and controllability. Extensive experiments on our proposed 3SGen-Bench and other public benchmarks demonstrate our superior performance across diverse image-driven generation tasks.

Figures (11)

• Figure 1: Versatile samples of the proposed 3SGen. Given one or multiple reference images along with a text prompt, 3SGen generates consistent high-fidelity results.
• Figure 2: The overall architecture of our proposed 3SGen. Semantic queries serve as resamplers that pass through the MLLM to obtain aligned multimodal features, which then interact with adaptive task-specific memory priors to derive task-aware semantic representations. VAE tokens from reference images are incorporated as joint inputs to supplement fine-grained visual details.
• Figure 3: Visualization of the hierarchical training strategy. Noise input is omitted for simplification.
• Figure 4: Showcases of samples from our proposed 3SGen-Bench.
• Figure 5: Qualitative comparison of subject-driven generation.