CovMatch: Cross-Covariance Guided Multimodal Dataset Distillation with Trainable Text Encoder

TL;DR

CovMatch addresses the scalability challenge of multimodal dataset distillation by enabling joint optimization of image and text encoders through cross-covariance alignment in a linearized contrastive learning setting. It replaces costly bi-level or trajectory-based optimization with a closed-form inner solution and combines cross-covariance matching with per-modality feature distribution regularization, supplemented by an online encoder update. Empirically, CovMatch achieves state-of-the-art retrieval on Flickr30K and COCO with as few as 500 synthetic image-text pairs and demonstrates strong cross-architecture generalization across NFNet, ViT, and various text encoders. The work highlights the importance of jointly training both modalities for semantic alignment and offers a scalable, practical framework for efficient vision-language learning.

Abstract

Multimodal dataset distillation aims to synthesize a small set of image-text pairs that enables efficient training of large-scale vision-language models. While dataset distillation has shown promise in unimodal tasks, extending it to multimodal contrastive learning presents key challenges: learning cross-modal alignment and managing the high computational cost of large encoders. Prior approaches address scalability by freezing the text encoder and update only the image encoder and text projection layer. However, we find this severely limits semantic alignment and becomes a bottleneck for performance scaling. We propose CovMatch, a scalable dataset distillation framework that aligns the cross-covariance of real and synthetic features while regularizing feature distributions within each modality. Unlike prior approaches, CovMatch enables joint optimization of both encoders, leading to stronger cross-modal alignment and improved performance. Evaluated on Flickr30K and COCO, CovMatch outperforms state-of-the-art multimodal distillation methods and achieves up to 6.8% absolute gains in retrieval accuracy using only 500 synthetic pairs.

Figures (8)

• Figure 1: Comparison of multimodal dataset distillation frameworks. Prior methods freeze the entire text encoder and pass distilled text only through a linear projection. In contrast, we freeze only the input embedding layer (i.e., token and position embedding module) and include the transformer layers in the distillation process.
• Figure 2: (a) T-SNE visualization of text embeddings from the Flickr30K test set. Models are trained with $N=500$ synthetic pairs distilled by LoRS (left) and CovMatch (right). (b) Intra- and inter-pair average similarity of test captions for models trained with $N=100$ and $500$ synthetic samples. (c) Retrieval performance as a function of the number of synthetic pairs $N$.
• Figure 3: Illustration of our proposed method, CovMatch. At each distillation step, CovMatch updates the synthetic image-text pairs using a matching loss computed with an alternately updated online model. The loss consists of two components: 1) Cross-covariance matching loss \ref{['L_cov']}, computed as the distance between the cross-covariance matrices of real and synthetic features (extracted before the projection layers); and 2) Feature matching loss \ref{['eqn:loss_feature_matching']}, computed as the distance between the mean feature vectors of real and synthetic data after the projection, applied independently to each modality.
• Figure 4: (a) Singular values of the cross-covariance matrix $C^\mathcal{T}$ computed on the full Flickr30K dataset, showing its low-rank structure. (b) Evolution of the cross-covariance matching loss $\mathcal{L}^\textrm{cov}$\ref{['L_cov']} and the feature matching loss $\mathcal{L}_v^\textrm{feat} + \mathcal{L}_l^\textrm{feat}$\ref{['eqn:loss_feature_matching']} during distillation with only the cross-covariance objective for 500 synthetic pairs.
• Figure 5: (a) Ablation on the scaling factor $\rho$ in \ref{['L_cov']}. Scaling the real cross-covariance $C^\mathcal{T}$ is particularly important when $N$ is small. (b) Ablation on the feature matching weight $\lambda$ in \ref{['eq:final_loss']}. The optimal value of $\lambda$ tends to increase as $N$ increases. (c) Ablation on the online model update strategy. Updating with real data exhibits the best performance. All ablation studies are conducted on Flickr30k with varying $N$.