EVLF: Early Vision-Language Fusion for Generative Dataset Distillation

TL;DR

An Early Vision-Language Fusion (EVLF) method that aligns textual and visual embeddings at the transition between the encoder and the generative backbone, yielding semantically faithful and visually coherent synthetic data, yielding consistent improvements in downstream classification accuracy across varied settings.

Abstract

Dataset distillation (DD) aims to synthesize compact training sets that enable models to achieve high accuracy with significantly fewer samples. Recent diffusion-based DD methods commonly introduce semantic guidance through late-stage cross-attention, where textual prompts tend to dominate the generative process. Although this strategy enforces label relevance, it diminishes the contribution of visual latents, resulting in over-corrected samples that mirror prompt patterns rather than reflecting intrinsic visual features. To solve this problem, we introduce an Early Vision-Language Fusion (EVLF) method that aligns textual and visual embeddings at the transition between the encoder and the generative backbone. By incorporating a lightweight cross-attention module at this transition, the early representations simultaneously encode local textures and global semantic directions across the denoising process. Importantly, EVLF is plug-and-play and can be easily integrated into any diffusion-based dataset distillation pipeline with an encoder. It works across different denoiser architectures and sampling schedules without any task-specific modifications. Extensive experiments demonstrate that EVLF generates semantically faithful and visually coherent synthetic data, yielding consistent improvements in downstream classification accuracy across varied settings. Source code is available at https://github.com/wenqi-cai297/earlyfusion-for-dd/.

Figures (5)

• Figure 1: Comparison between traditional late-fusion approaches and the proposed EVLF. (a) Late-fusion methods inject textual prompts during the denoising process, causing semantic signals to dominate visual latent representations. (b) EVLF introduces vision-language alignment before diffusion, allowing semantic cues and visual features to co-evolve throughout generation. (c) Synthetic samples on ImageNette (256 $\times$ 256). (d) Synthetic samples on CIFAR-10 (32 $\times$ 32). Rows display real images, late-fusion results, and EVLF results. EVLF produces samples with stronger label fidelity and more coherent visual details.
• Figure 2: Overview of EVLF. Visual latents from a VAE and text embeddings from a text encoder are fused via cross-attention at the encoder-backbone interface. The fused embeddings are trained with $\mathcal{L}_{\mathrm{MSE}}$ for visual preservation and $\mathcal{L}_{\mathrm{InfoNCE}}$ for semantic alignment. Fused embeddings are clustered and decoded to produce the distilled synthetic dataset.
• Figure 3: t-SNE visualization of synthetic and real samples on four ImageNet-1K classes. D$^4$M su2024d and MGD$^3$chan2025mgd produce synthetic samples that occupy limited regions of the real-data manifold. With EVLF, the synthesized samples cover a broader and more varied region, indicating improved diversity and distributional alignment.
• Figure 4: Parameter Analysis of $\lambda_1$ on ImageIDC.
• Figure 5: Visualization of synthesized images generated by D$^4$M and our EVLF under low- and high-resolution settings. (a) Bird class from CIFAR-10, and (b) Beagle class from ImageNet-1K. EVLF produces samples with clearer structure, richer textures, and improved consistency with class semantics across different image scales.