SSR-Encoder: Encoding Selective Subject Representation for Subject-Driven Generation

TL;DR

SSR-Encoder introduces a finetuning-free selective subject encoder for zero-shot subject-driven generation, capable of handling single or multiple subjects from reference images via text or mask queries. It combines a Token-to-Patch Aligner and a Detail-Preserving Subject Encoder to create multi-scale subject embeddings and inject them through trainable cross-attention layers into diffusion models, optimized with $L_{LDM}$ and embedding consistency loss $L_{reg}$. The approach achieves state-of-the-art performance among finetuning-free methods and competes with finetuning-based techniques on benchmarks like the Multi-Subject Bench and DreamBench, delivering high fidelity, editability, and controllability, while remaining compatible with ControlNet and AnimateDiff. These results advance practical personalized image generation by enabling precise subject representation selection, improving efficiency and generalization across diverse models and control modalities.

Abstract

Recent advancements in subject-driven image generation have led to zero-shot generation, yet precise selection and focus on crucial subject representations remain challenging. Addressing this, we introduce the SSR-Encoder, a novel architecture designed for selectively capturing any subject from single or multiple reference images. It responds to various query modalities including text and masks, without necessitating test-time fine-tuning. The SSR-Encoder combines a Token-to-Patch Aligner that aligns query inputs with image patches and a Detail-Preserving Subject Encoder for extracting and preserving fine features of the subjects, thereby generating subject embeddings. These embeddings, used in conjunction with original text embeddings, condition the generation process. Characterized by its model generalizability and efficiency, the SSR-Encoder adapts to a range of custom models and control modules. Enhanced by the Embedding Consistency Regularization Loss for improved training, our extensive experiments demonstrate its effectiveness in versatile and high-quality image generation, indicating its broad applicability. Project page: https://ssr-encoder.github.io

Figures (17)

• Figure 1: Our SSR-Encoder is a model generalizable encoder, which is able to guide any customized diffusion models for single subject-driven image generation (top branch) or multiple subject-driven image generation from different images (middle branch) based on the image representation selected by the text query or mask query without any additional test-time finetuning. Furthermore, our SSR-Encoder can also be applied for the controllable generation with additional control (bottom branch).
• Figure 2: Overall schematics of our method. Given a query text-image pairs $\left(q, I\right)$, the SSR-Encoder employs a token-to-patch aligner to highlight the selective regions in the reference image by the query. It extracts more fine-grained details of the subject through the detail-preserving subject encoder, projecting multi-scale visual embeddings via the token-to-patch aligner. Then, we adopt subject-conditioned generation to generate specific subjects with high fidelity and creative editability. During training, we adopt reconstruction loss $L_{LDM}$ and embedding consistency regularization loss $L_{reg}$ for selective subject-driven learning.
• Figure 3: Qualitative results of SSR-Encoder in different generative capabilities. Our method supports two query modalities and is adaptable for a variety of tasks, including single- and multi-subject conditioned generation. Its versatility extends to integration with other customized models and compatibility with off-the-shelf ControlNets.
• Figure 4: Qualitative comparison of different methods. Our results not only excel in editability and exclusivity but also closely resemble the reference subjects in visual fidelity. Notably, the SSR-Encoder achieves this without the need for fine-tuning.
• Figure 5: Visualization of attention maps $A_{t2p}$.