Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

On the Cone Effect and Modality Gap in Medical Vision-Language Embeddings

David Restrepo, Miguel L Martins, Chenwei Wu, Luis Filipe Nakayama, Diego M Lopez, Stergios Christodoulidis, Maria Vakalopoulou, Enzo Ferrante

Abstract

Vision-Language Models (VLMs) exhibit a characteristic "cone effect" in which nonlinear encoders map embeddings into highly concentrated regions of the representation space, contributing to cross-modal separation known as the modality gap. While this phenomenon has been widely observed, its practical impact on supervised multimodal learning -particularly in medical domains- remains unclear. In this work, we introduce a lightweight post-hoc mechanism that keeps pretrained VLM encoders frozen while continuously controlling cross-modal separation through a single hyperparameter {λ}. This enables systematic analysis of how the modality gap affects downstream multimodal performance without expensive retraining. We evaluate generalist (CLIP, SigLIP) and medically specialized (BioMedCLIP, MedSigLIP) models across diverse medical and natural datasets in a supervised multimodal settings. Results consistently show that reducing excessive modality gap improves downstream performance, with medical datasets exhibiting stronger sensitivity to gap modulation; however, fully collapsing the gap is not always optimal, and intermediate, task-dependent separation yields the best results. These findings position the modality gap as a tunable property of multimodal representations rather than a quantity that should be universally minimized.

On the Cone Effect and Modality Gap in Medical Vision-Language Embeddings

Abstract

Vision-Language Models (VLMs) exhibit a characteristic "cone effect" in which nonlinear encoders map embeddings into highly concentrated regions of the representation space, contributing to cross-modal separation known as the modality gap. While this phenomenon has been widely observed, its practical impact on supervised multimodal learning -particularly in medical domains- remains unclear. In this work, we introduce a lightweight post-hoc mechanism that keeps pretrained VLM encoders frozen while continuously controlling cross-modal separation through a single hyperparameter {λ}. This enables systematic analysis of how the modality gap affects downstream multimodal performance without expensive retraining. We evaluate generalist (CLIP, SigLIP) and medically specialized (BioMedCLIP, MedSigLIP) models across diverse medical and natural datasets in a supervised multimodal settings. Results consistently show that reducing excessive modality gap improves downstream performance, with medical datasets exhibiting stronger sensitivity to gap modulation; however, fully collapsing the gap is not always optimal, and intermediate, task-dependent separation yields the best results. These findings position the modality gap as a tunable property of multimodal representations rather than a quantity that should be universally minimized.
Paper Structure (17 sections, 7 equations, 3 figures, 1 table)

This paper contains 17 sections, 7 equations, 3 figures, 1 table.

Table of Contents

  1. Introduction
  2. Methods
  3. Problem Statement: Cone Effect and Modality Gap in Vision--Language Models
  4. Modality Gap in Vision--Language Embedding Spaces.
  5. Cone Effect in Vision--Language Embedding Spaces.
  6. Cone effect in Medical Domains.
  7. Cross-domain Embedding Alignment Intervention.
  8. Experimental Setup
  9. Vision--Language Backbones
  10. Datasets and Domain Shift Regimes
  11. Embedding Alignment Protocol
  12. Linear Probe Downstream Evaluation
  13. Evaluation Metrics and Statistical Protocol
  14. Modality Gap Visualization
  15. Results and Discussion
  16. ...and 2 more sections

Figures (3)

  • Figure 1: Overview of the proposed framework. (A) Embedding alignment pipeline: raw image and text features are extracted, shifted by a hyperparameter $\lambda$ to modulate the modality gap, and evaluated via linear probing. (B) Overview of medical datasets and corresponding multimodal tasks used in this study.
  • Figure 2: Low-dimensional visualization of image and text embeddings across datasets and backbones. Projections are shown using the first two principal components in three-dimensional ambient space.
  • Figure 3: Downstream AUC as a function of alignment strength $\lambda$ across datasets and backbones. Moderate reduction of the modality gap consistently improves performance, while excessive closure can lead to saturation or degradation, particularly for medical-domain models. Error bars denote standard deviation across seeds.