Non-Contrastive Vision-Language Learning with Predictive Embedding Alignment

TL;DR

We address the fragility of contrastive vision-language pretraining in data-scarce medical settings by introducing NOVA, a non-contrastive framework that aligns image embeddings to a frozen domain-specific text space through embedding prediction and distributional regularization. The method relies on a simple objective combining a prediction loss $\\mathcal{L}_{\text{MSE}}$ and Sketched Isotropic Gaussian Regularization $\\mathcal{L}_{\text{SIGReg}}$, controlled by a single hyperparameter $\lambda$, and interprets the learning process via an energy-based viewpoint with $E(V,T)=\| P_V - E_T\|_2^2$. Trained from scratch with ViT encoders and a frozen ClinicalBERT target, NOVA achieves state-of-the-art zero-shot chest X-ray classification across MIMIC-CXR, ChestX-ray14, and CheXpert, while displaying far more stable training and requiring less hyperparameter tuning than contrastive baselines. The results highlight the practicality and robustness of non-contrastive vision-language learning in medical domains, suggesting extensions to other modalities and potential domain-specific text-model adaptations to further close the semantic gap between clinical imagery and language.

Abstract

Vision-language models have transformed multimodal representation learning, yet dominant contrastive approaches like CLIP require large batch sizes, careful negative sampling, and extensive hyperparameter tuning. We introduce NOVA, a NOn-contrastive Vision-language Alignment framework based on joint embedding prediction with distributional regularization. NOVA aligns visual representations to a frozen, domain-specific text encoder by predicting text embeddings from augmented image views, while enforcing an isotropic Gaussian structure via Sketched Isotropic Gaussian Regularization (SIGReg). This eliminates the need for negative sampling, momentum encoders, or stop-gradients, reducing the training objective to a single hyperparameter. We evaluate NOVA on zeroshot chest X-ray classification using ClinicalBERT as the text encoder and Vision Transformers trained from scratch on MIMIC-CXR. On zero-shot classification across three benchmark datasets, NOVA outperforms multiple standard baselines while exhibiting substantially more consistent training runs. Our results demonstrate that non-contrastive vision-language pretraining offers a simpler, more stable, and more effective alternative to contrastive methods.

Figures (4)

• Figure 1: NOVA overview. Given an image-text pair, we generate $n$ augmented views of the image via global and local crops. Each view is encoded by a learnable vision encoder and passed through a learnable predictor to obtain predicted embeddings $\{P_1, \ldots, P_n\}$. The corresponding text is encoded by a frozen, pretrained ClinicalBERT model with a learnable projection head, yielding the target embedding $E_T$. All predicted image views are aligned to the text anchor via MSE loss, while SIGReg regularization enforces an isotropic Gaussian distribution over the joint embedding space to prevent representational collapse. This simple objective enables stable training without negative sampling, momentum encoders, or stop-gradients.
• Figure 2: ChestX-ray14 per-pathology results. NOVA outperforms baselines across multiple conditions, with particularly strong gains on Atelectasis and Effusion.
• Figure 3: CheXpert per-pathology results. NOVA achieves strong performance on Atelectasis and Consolidation, leveraging its fine-grained vision capabilities.
• Figure 4: Training dynamics. NOVA converges smoothly across seeds, while MedCLIP-style training overfits after approximately 10 epochs.