EchoDistill: Bidirectional Concept Distillation for One-Step Diffusion Personalization

TL;DR

EchoDistill tackles one-step diffusion personalization (1-SDP) by training a multi-step teacher and a one-step student in an end-to-end, bidirectional framework. A shared text encoder ensures consistent semantic grounding, while a distillation stage aligns the student to the teacher through alignment and adversarial losses. The novel echoing stage lets the fast student generate samples that feedback to improve both models, enhancing concept fidelity and generation quality. Extensive DreamBench experiments show EchoDistill outperforms existing 1-SDP methods in both identity preservation and image-text alignment, with efficient inference suitable for real-time use. This work establishes a new paradigm for rapid, reliable concept personalization in diffusion-based text-to-image generation.

Abstract

Recent advances in accelerating text-to-image (T2I) diffusion models have enabled the synthesis of high-fidelity images even in a single step. However, personalizing these models to incorporate novel concepts remains a challenge due to the limited capacity of one-step models to capture new concept distributions effectively. We propose a bidirectional concept distillation framework, EchoDistill, to enable one-step diffusion personalization (1-SDP). Our approach involves an end-to-end training process where a multi-step diffusion model (teacher) and a one-step diffusion model (student) are trained simultaneously. The concept is first distilled from the teacher model to the student, and then echoed back from the student to the teacher. During the EchoDistill, we share the text encoder between the two models to ensure consistent semantic understanding. Following this, the student model is optimized with adversarial losses to align with the real image distribution and with alignment losses to maintain consistency with the teacher's output. Furthermore, we introduce the bidirectional echoing refinement strategy, wherein the student model leverages its faster generation capability to feedback to the teacher model. This bidirectional concept distillation mechanism not only enhances the student ability to personalize novel concepts but also improves the generative quality of the teacher model. Our experiments demonstrate that this collaborative framework significantly outperforms existing personalization methods over the 1-SDP setup, establishing a novel paradigm for rapid and effective personalization in T2I diffusion models.

Figures (12)

• Figure 1: Comparison with existing new concept learning methods for one-step personalization: Textual Inversion textual_inversion and Custom Diffusion kumari2023customdiffusion for SDTurbo sauer2023adversarial, and IP-Adapter ye2023ip-adapter for TCD zheng2024trajectory_tcd.
• Figure 2: Overview of EchoDistill. The student and teacher jointly learn the new concept with a shared text encoder. The teacher learns from real images $x_0^r$ (green line), and the text encoder is updated accordingly. The student is optimized with two objectives (gold line): an adversarial loss to match real data distribution and alignment losses to match the denoised outputs of the teacher. The discriminators are trained to distinguish between the student's outputs and real images.
• Figure 3: Our method EchoDistill (last column) compared with existing methods applied to the 1-SDP setup with SDTurbo sauer2023adversarial as the one-step diffusion backbone. One representive concept image is shown on the left-most column.
• Figure 4: (a) Abating the Infer-NFEs; (b) Ablating the one-step diffusion backbone; (c) Ablating the teacher and discriminators; (d) Ablating the echoing stage.
• Figure 5: The qualitative results of the 1-shot performance.