Unified Vision-Language Modeling via Concept Space Alignment

TL;DR

V-LCM matches state-of-the-art vision-language models on tasks covering image/video captioning and question answering, while significantly outperforming them across 61 rich- to low-resource languages out of all 62 tested languages.

Abstract

We introduce V-SONAR, a vision-language embedding space extended from the text-only embedding space SONAR (Omnilingual Embeddings Team et al., 2026), which supports 1500 text languages and 177 speech languages. To construct V-SONAR, we propose a post-hoc alignment pipeline that maps the representations of an existing vision encoder into the SONAR space. We thoroughly evaluate V-SONAR and show that its embeddings achieve competitive performance on text-to-video retrieval. Equipped with the OMNISONAR text decoder, V-SONAR further surpasses state-of-the-art vision-language models on video captioning tasks, including DREAM-1K (BLEU 23.9 vs. 19.6) and PE-VIDEO (BLEU 39.0 vs. 30.0). Leveraging V-SONAR, we first demonstrate that the Large Concept Model (LCM; LCM team et al. 2024) operating in SONAR and trained with English text only, can perform both single- and multi-visual concept understanding in a zero-shot manner. Finally, we introduce V-LCM, which extends the LCM with vision-language instruction tuning. V-LCM encodes vision and language inputs into an unified sequence of latent embeddings via V-SONAR and SONAR, and it is trained with the same latent diffusion objective for next-embedding prediction as in LCM's text-only pre-training. Experiments on a large-scale multilingual and -modal instruction-tuning data mixture highlight the potential of V-LCM: V-LCM matches state-of-the-art vision-language models on tasks covering image/video captioning and question answering, while significantly outperforming them across 61 rich- to low-resource languages out of all 62 tested languages.

Figures (21)

• Figure 1: Left: Illustration of v-Sonar. Right: training v-LCM with vision-language instruction tuning.
• Figure 2: Comparison for v-Sonar trained with SONAR version 1 and 2 (OmniSONAR) embedding space on three captioning datasets.
• Figure 3: Operating in v-Sonar space, the LCM performs better than only accepting the textual Sonar inputs. We compare LCM-7B-IFT in VideoXum with Rouge-L scores across short, mid, and long categories of video inputs.
• Figure 4: Performance in 62 languages for v-LCM, Qwen2.5-VL-7B and PLM-8B on M3IT testing set for MSRVTT, MSRVTT-QA, ImageNet, VQA-V2, VIST and OKVQA. We report the Rouge-L scores averaged from all datasets. Detailed results for each dataset can be found in the \ref{['appendix:detailed-m3it-multilinguaul']}.
• Figure 5: M3IT evaluation on 61 languages for MSRVTT.