A Concept-Centric Approach to Multi-Modality Learning

TL;DR

This work introduces a concept-centric framework for multi-modality learning built around a modality-agnostic concept space composed of probabilistic box embeddings and a set of modality-specific projection models that map inputs into a shared knowledge space. By grounding cross-modal representations in explicit concept entailments, the approach achieves efficient learning, strong cross-modality alignment without joint training, and easy incorporation of new modalities. It demonstrates competitive performance on Image-Text Matching and Visual Question Answering while emphasizing interpretability and modularity over raw task-specific optimization. The results suggest a promising direction toward cognitively inspired AI systems that reuse abstract knowledge across modalities and tasks, with scalable paths toward richer concept spaces and broader modalities.

Abstract

Humans possess a remarkable ability to acquire knowledge efficiently and apply it across diverse modalities through a coherent and shared understanding of the world. Inspired by this cognitive capability, we introduce a concept-centric multi-modality learning framework built around a modality-agnostic concept space that captures structured, abstract knowledge, alongside a set of modality-specific projection models that map raw inputs onto this shared space. The concept space is decoupled from any specific modality and serves as a repository of universally applicable knowledge. Once learned, the knowledge embedded in the concept space enables more efficient adaptation to new modalities, as projection models can align with existing conceptual representations rather than learning from scratch. This efficiency is empirically validated in our experiments, where the proposed framework exhibits faster convergence compared to baseline models. In addition, the framework's modular design supports seamless integration of new modalities, since projection models are trained independently yet produce unified outputs within the shared concept space. We evaluate the framework on two representative downstream tasks. While the focus is not on task-specific optimization, the framework attains comparable results with a smaller training footprint, no task-specific fine-tuning, and inference performed entirely within a shared space of learned concepts that offers interpretability. These findings point toward a promising direction for developing learning systems that operate in a manner more consistent with human cognitive processes.

Figures (10)

• Figure 1: Overall structure of the proposed concept-centric multi-modality learning framework. A modality-agnostic concept space is trained to reflect the relations between the set of concepts $\mathcal{Y}$ as observed in a training dataset $\mathcal{D}$ (left). Modality-specific projection models are trained to create projections $\Omega$ for their inputs based on the inputs’ associations with concepts (middle). The modular design of the framework offers great flexibility and adaptability to a wide range of downstream tasks (right).
• Figure 2: Learning curves of the proposed projection models and the baseline model. The plot displays the evaluation accuracy on category concepts and the evaluation mAP on attribute concepts measured every 50 training steps. During the learning process, the proposed vision-modality projection model improves more quickly compared to the baseline thanks to the universal concept space that already has abstract knowledge embedded in it. This faster learning process of our framework bridges the efficiency gap between traditional machine learning methods, which require a large amount of data, and human learning that excels at extracting modality-specific representations and linking them to structured abstract knowledge.
• Figure 3: Comparison of training efficiency for cross-modality alignment. Validation accuracy for image-text matching task of the Unified dataset is used as a proxy for evaluating cross-modality alignment. Our proposed method reaches near-saturated alignment without finetuning and attains its final accuracy within a significantly shorter period, while ViLT, CLIP, and FLAVA require substantially more computation time to achieve comparable alignment.
• Figure 4: Modality alignment measured by entailment probabilities between positive and negative cross-modality representation pairs across image, English, Chinese, Spanish, and French modalities. Left: higher values (↑) indicate stronger projection overlap for positive concept pairs. Right: lower values (↓) indicate stronger separation for negative pairs. All modality-specific models are trained independently.
• Figure 5: A comparison between the learned concept space's understanding of the CLEVR world and the ground truth relations illustrated via entailment probabilities of concept pairs. Such comparison allows simple probing into the knowledge learned by this abstract concept space. A SoftMax function is applied on entailment probabilities of same-attribute concepts conditioned on a single concept $y$ so $\sum_{y^{\prime} \in \text{attr}_i} P(y^{\prime}|y) = 1$ is satisfied.