Model alignment using inter-modal bridges

TL;DR

This work tackles cross-modality reuse by learning inter-modal bridges that align latent spaces with minimal supervision. It combines conditional flow matching (CFM) and entropic optimal transport (OT) under a novel inter-modal bridge cost to morph representations across modalities, using true, global, or local alignment strategies. Across image-text and brain–artificial neural pairings, the approach achieves competitive downstream performance with less than 20% paired data, and demonstrates that a global OT alignment helps prevent overfitting in noisy settings. The results highlight the practical potential of data-efficient, cross-modal alignment for object recognition and image synthesis, while offering a principled framework for extending alignments to diverse domains.

Abstract

Foundation models have demonstrated remarkable performance across modalities such as language and vision. However, model reuse across distinct modalities (e.g., text and vision) remains limited due to the difficulty of aligning internal representations. Existing methods require extensive paired training data or are constrained to specific domains. We introduce a semi-supervised approach for model alignment via conditional flow matching. The conditional flow between latent spaces of different modalities (e.g., text-to-image or biological-to-artificial neuronal activity) can be learned in two settings: ($1$) solving a (balanced or unbalanced) optimal transport problem with an inter-space bridge cost, and ($2$) performing memory-efficient alignment using labelled exemplars. Despite being constrained by the original models' capacity, our method--under both settings--matches downstream task performance of end-to-end trained models on object recognition and image generation tasks across MNIST, ImageNet, and \cite{majaj2015simple} datasets, particularly when labelled training data is scarce ($<20\%$). Our method provides a data-efficient solution for inter-modal model alignment with minimal supervision.

Figures (24)

• Figure 1: Pictorial representation of our approach for aligning model space using inter-modal bridges. We consider two pre-trained models; $f_\mathcal{X}$ and $f_\mathcal{Y}$ and their corresponding datasets $D_\mathcal{X}$ and $D_\mathcal{Y}$. Next, we obtain source and target latent distributions $\mu$ and $\nu$ and compute the optimal coupling $\pi^\star$ across these two distributions using paired samples $(x_i^p,y_j^p) \in P$ as inter-space bridges or using the paired sampled directly. Given this, we learn the velocity field $v_{t,\theta}$ that morphs noise $\rho$ conditioned on $x_i$ in the source distribution to some target $y_i$ using samples from the optimal coupling $\pi^\star(\cdot \mid x_i)$.
• Figure 2: Pictorial representation of alignment methods. Here, --- represents true pairs, $x_{ip}, y_{ip} \in P$, $-~-$ the coupling given some inter-space cost $C_{XY}$ and OT solver. $\circ$, $\circ$ represent sample batches.
• Figure 3: Pictorial representation of bridge cost via $(x_{i}^p,y_{j}^p) \in P$ and $(x_{i},y_{j}) \not\in P$.
• Figure 4: Noise distribution trajectory to the target latent space of a language model at $t=(0,0.5,1)$ using true alignment with $1\%$ paired points. The latent source and target feature spaces for ImageNet are visualised using UMAP mcinnes2018umap. Top: Classes with minimal overlap in the image latent space. Bottom: Classes with high overlap in the image latent space.
• Figure 5: Latent space overlaps on conditional flow matching from image-to-text domain, using true alignment in a fully supervised setting. A) MNIST experiment. B) ImageNet experiment. C) Relationship between the number of classes and the degree of overlap in the latent space for the ImageNet dataset. D) UMAP visualisations of the classes with the highest overlap in the latent space of the ViT-B model.