BrokenBind: Universal Modality Exploration beyond Dataset Boundaries

TL;DR

BrokenBind addresses the core challenge of modality binding when the target modalities are not simultaneously present in any single dataset. It introduces modality extrapolation (MOX) to generate pseudo embeddings by leveraging cross-modal and cross-dataset transitions via pivot modalities, formalized through transition matrices such as $\mathbf{W}^{b\text{-}c}=\mathbf{F}^{c2}{\mathbf{F}^{b2}}^{+}$ and $\mathbf{W}^{2\text{-}1}=\mathbf{F}^{b1}{\mathbf{F}^{b2}}^{+}$, all regularized by a Frobenius term. The objective combines MOX with CyCLIP losses to enforce cross-data and cross-modal consistency, yielding the joint objective $\mathcal{L}=\mathcal{L}_{\text{MOX}}+\mathcal{L}_{\text{CyCLIP}}$. Extensive experiments across two- and three-dataset settings, with backbones from ImageBind, LanguageBind, Video-CLIP, CLAP, and TVL, demonstrate that BrokenBind substantially surpasses baselines in both accuracy (mAP) and emergent modality binding, including zero-shot and unseen-modality cases. The work highlights practical utility for universal modality exploration with reduced data collection costs, while acknowledging limitations in rare-modality scenarios and proposing directions for balancing modality availability in future work.

Abstract

Multi-modal learning combines various modalities to provide a comprehensive understanding of real-world problems. A common strategy is to directly bind different modalities together in a specific joint embedding space. However, the capability of existing methods is restricted within the modalities presented in the given dataset, thus they are biased when generalizing to unpresented modalities in downstream tasks. As a result, due to such inflexibility, the viability of previous methods is seriously hindered by the cost of acquiring multi-modal datasets. In this paper, we introduce BrokenBind, which focuses on binding modalities that are presented from different datasets. To achieve this, BrokenBind simultaneously leverages multiple datasets containing the modalities of interest and one shared modality. Though the two datasets do not correspond to each other due to distribution mismatch, we can capture their relationship to generate pseudo embeddings to fill in the missing modalities of interest, enabling flexible and generalized multi-modal learning. Under our framework, any two modalities can be bound together, free from the dataset limitation, to achieve universal modality exploration. Further, to reveal the capability of our method, we study intensified scenarios where more than two datasets are needed for modality binding and show the effectiveness of BrokenBind in low-data regimes. Through extensive evaluation, we carefully justify the superiority of BrokenBind compared to well-known multi-modal baseline methods.

Figures (8)

• Figure 1: Example of BrokenBind and previous binding strategy for fetching objects in a room. The three modalities tactile sensing ("ta"), vision ("vi"), and point cloud ("po") are from space $\mathbb{M}^{ta}$, $\mathbb{M}^{vi}$, and $\mathbb{M}^{po}$, respectively, and are further correspondingly encoded by encoders $g^{ta}$, $g^{vi}$, and $g^{po}$ into a joint space (denoted by different colors). Existing approaches require multi-modal correspondence which is hard to satisfy, instead, BrokenBind aims to leverage multiple datasets effectively by bridging the dataset gap.
• Figure 2: Illustration of interpolation, extrapolation, and multi-extrapolation. Different colors denote different examples. The interpolation shows the knowledge in two known examples, but the extrapolation show emerging knowledge.
• Figure 3: Illustration of cross-modal cross-data consistency. The superscripts denote modality and dataset, subscripts $i$, $j$, $k$, and $l$ are indices, e.g., $x_i^{a1}$ indicates the $i$-th datum from modality $a$ in $\mathcal{D}^1$. Lines with "$=$" denote that consistency is applied. The arrows denote the transition relationships from one example to another. Left: Before enforcing consistency, the modality and data structure are inconsistent. Right: After enforcing consistency, a more symmetric structure is presented, benefiting the generalization to novel data and modalities.
• Figure 4: Left: Efficiency study between BrokenBind and LoRA fine-tuning. Right: Ablation study on different module settings.
• Figure 5: t-SNE visualization of embeddings. Left: modality extrapolation (MOX) on text. Right: ground truth (GT) embeddings.