Capybara-OMNI: An Efficient Paradigm for Building Omni-Modal Language Models

TL;DR

Capybara-OMNI tackles the data and computation bottlenecks of building omni-modal language models by proposing a lightweight, three-stage training pipeline that sequentially aligns visual and audio modalities before cross-modal instruction tuning. The architecture leverages a SigLIP-based visual encoder, a compact MLP adaptor, and a LLM (Qwen-derived) to fuse text, image, video, and audio into a single model, while freezing/unfreezing strategies mitigate catastrophic forgetting across stages. Through substantial but curated training data and targeted data quality controls, Capybara-OMNI achieves competitive image, video, and audio understanding across open benchmarks, and its cross-modal tuning yields improved interaction and conversational capabilities. The work demonstrates that full-modality capabilities can be realized with modest data and compute and contributes an open-source release that supports further research and development in multimodal interaction.

Abstract

With the development of Multimodal Large Language Models (MLLMs), numerous outstanding accomplishments have emerged within the open-source community. Due to the complexity of creating and training multimodal data pairs, it is still a computational and time-consuming process to build powerful MLLMs. In this work, we introduce Capybara-OMNI, an MLLM that trains in a lightweight and efficient manner and supports understanding text, image, video, and audio modalities. We present in detail the framework design, the data construction, and the training recipe, to develop an MLLM step-by-step to obtain competitive performance. We also provide exclusive benchmarks utilized in our experiments to show how to properly verify understanding capabilities across different modalities. Results show that by following our guidance, we can efficiently build an MLLM that achieves competitive performance among models of the same scale on various multimodal benchmarks. Additionally, to enhance the multimodal instruction following and conversational capabilities of the model, we further discuss how to train the chat version upon an MLLM understanding model, which is more in line with user habits for tasks like real-time interaction with humans. We publicly disclose the Capybara-OMNI model, along with its chat-based version. The disclosure includes both the model weights, a portion of the training data, and the inference codes, which are made available on GitHub.

Figures (6)

• Figure 1: The architecture of Capybara-OMNI. It supports various modalities as input, including text, image, video and audio. Moreover, it expands the output modalities to text and audio.
• Figure 2: The overview of training Capybara-OMNI. We propose to handle multiple modalities in a progressive manner, which consists of three main stages. We start from the alignment between image and text and then expand to other modalities. In each stage, we freeze (i.e., the blue snow symbol) or unfreeze (i.e., the red fire symbol) the module parameters to update the model.
• Figure 3: The training data construction process for Capybara-OMNI.
• Figure 4: Prompt for generating single-round dialogues based on video content.
• Figure 5: Prompt for generating multi-round dialogues based on video content.