Modest-Align: Data-Efficient Alignment for Vision-Language Models

TL;DR

Modest-Align tackles cross-modal alignment under data- and compute-constrained settings by moderating overconfidence and robustly handling weakly aligned image-text pairs. It introduces Random Perturbation to simulate uncertainty and Embedding Smoothing to relax targets, integrated into a Confidence-Calibrated Contrastive Loss with lightweight Vision-Language adapters built on frozen unimodal encoders. Empirically, it achieves competitive retrieval performance with about 3.5M training pairs—100x less data and roughly 600x less training time than CLIP on MS-COCO—while outperforming state-of-the-art baselines on several benchmarks. The approach demonstrates strong generalization, especially on noisier datasets, and offers a scalable solution for real-world, low-resource cross-modal alignment.

Abstract

Cross-modal alignment aims to map heterogeneous modalities into a shared latent space, as exemplified by models like CLIP, which benefit from large-scale image-text pretraining for strong recognition capabilities. However, when operating in resource-constrained settings with limited or low-quality data, these models often suffer from overconfidence and degraded performance due to the prevalence of ambiguous or weakly correlated image-text pairs. Current contrastive learning approaches, which rely on single positive pairs, further exacerbate this issue by reinforcing overconfidence on uncertain samples. To address these challenges, we propose Modest-Align, a lightweight alignment framework designed for robustness and efficiency. Our approach leverages two complementary strategies -- Random Perturbation, which introduces controlled noise to simulate uncertainty, and Embedding Smoothing, which calibrates similarity distributions in the embedding space. These mechanisms collectively reduce overconfidence and improve performance on noisy or weakly aligned samples. Extensive experiments across multiple benchmark datasets demonstrate that Modest-Align outperforms state-of-the-art methods in retrieval tasks, achieving competitive results with over 100x less training data and 600x less GPU time than CLIP. Our method offers a practical and scalable solution for cross-modal alignment in real-world, low-resource scenarios.

Figures (12)

• Figure 1: (a) Compared with CLIP and Fusemix, which assume full alignment during training (e.g., “Green leaves on many trees” is treated as a fully matched pair), our Modest-Align handles partially matched or noisy positive pairs by encouraging semi-aligned representations through tailored perturbation and smoothing strategies. (b) On the MS-COCO test dataset (Image → Text retrieval), Modest-Align outperforms both CLIP and Fusemix, despite using only 3.5M training samples—exceeding Fusemix trained on 5M samples (including our 3.5M subset) and CLIP trained on 400M pairs.
• Figure 2: (a-c) Three Paradigms for Cross-modal Alignment: CLIP for contrastive learning, FuseMix for embedding-level mixing, and Modest-Align for moderating overconfidence and enhancing robustness. Both FuseMix and Modest-Align derive positive pairs from mixed features, while Modest-Align further injects random perturbations into visual and textual embeddings to simulate uncertainty. “Positive” and “Negative” denote matched and mismatched pairs, with the gray region indicating uncertainty-aware perturbation. (d–f) Examples of ambiguous samples, including partially matched pairs (d, e) and completely mismatched pairs (f).
• Figure 3: A pipeline of the Modest-Align showcases the process of aligning the latent spaces of pre-trained unimodal encoders using a fewer dataset of paired data. The unimodal encoders remain frozen, with their latent encodings pre-computed only once for efficiency. In this framework, both Random Perturbation and Embedding Smoothing are applied to each latent space to enhance robustness and reduce model overconfidence. Lightweight V-L adapters are trained to meticulously align these augmented latents into a cohesive, shared latent space, effectively bridging the semantic gap between different modalities.
• Figure 4: Performance of Modest-Align and FuseMix vouitsis2024data on Flickr30K (a, c) and MS-COCO (b, d) test sets. Evaluated across training sets of varying scale (COCO for (a-b)), SBU for (c-d)), Modest-Align consistently demonstrates strong generalization and achieves SoTA performance on image retrieval benchmarks. Red indicates average improvement.
• Figure 5: (a) In the original distribution, clusters of image-text pairs are tightly packed, showing high model confidence. After implementing Modest-Align, the embeddings become more dispersed, indicating reduced confidence in individual labels and a more robust, softer probability distribution that enhances model generalization. (b) The ablation study of two techniques (RP and ES) in Recall@1.