BREEN: Bridge Data-Efficient Encoder-Free Multimodal Learning with Learnable Queries

TL;DR

BREEN tackles the data-hungry nature of encoder-free multimodal LLMs by distilling visual semantics from a pretrained encoder through learnable queries placed between image and text tokens, supervised by CLIP representations. An image expert in the FFN decouples image and query processing, boosting alignment without compromising the LLM’s textual reasoning. Through a three-stage training pipeline—pre-aligning, pretraining, and supervised fine-tuning—BREEN achieves competitive performance with only about 13 million text-image pairs, outperforming prior encoder-free models and narrowing the gap with encoder-based systems. The work demonstrates that structured knowledge transfer from a fixed vision encoder, combined with multi-granularity queries, offers a practical path to data-efficient, scalable multimodal learning.

Abstract

Encoder-free multimodal large language models(MLLMs) eliminate the need for a well-trained vision encoder by directly processing image tokens before the language model. While this approach reduces computational overhead and model complexity, it often requires large amounts of training data to effectively capture the visual knowledge typically encoded by vision models like CLIP. The absence of a vision encoder implies that the model is likely to rely on substantial data to learn the necessary visual-semantic alignments. In this work, we present BREEN, a data-efficient encoder-free multimodal architecture that mitigates this issue. BREEN leverages a learnable query and image experts to achieve comparable performance with significantly less training data. The learnable query, positioned between image and text tokens, is supervised by the output of a pretrained CLIP model to distill visual knowledge, bridging the gap between visual and textual modalities. Additionally, the image expert processes image tokens and learnable queries independently, improving efficiency and reducing interference with the LLM's textual capabilities. BREEN achieves comparable performance to prior encoder-free state-of-the-art models like Mono-InternVL, using only 13 million text-image pairs in training about one percent of the data required by existing methods. Our work highlights a promising direction for data-efficient encoder-free multimodal learning, offering an alternative to traditional encoder-based approaches.

Figures (7)

• Figure 1: Comparison of encoder-based and encoder-free multimodal large language models (MLLMs) in terms of pretraining data volume, average accuracy on evaluation benchmarks, and model size. The x-axis represents the amount of pretraining data (in millions) on a logarithmic scale, while the y-axis shows the average accuracy (%). The size of each circle corresponds to the model’s LLM backbone size.
• Figure 2: Comparison among different architectures. For Encoder-based MLLMs, we choose the series of LLaVA Model:LLaVAModel:LLaVA-1.5Model:LLaVA-NeXT as the representative. It feeds the images to the well-trained vision encoder, and project the outputs into the input space of the base LLM. For Encoder-free MLLMs, we choose previous works as EVE Model:EVE and Mono-InternVL Model:Mono-InternVL to compare. Instead of encoding the images with a vision encoder, these models use simpler and shallow patch embedding layers to encode the image patch directly.
• Figure 3: Overview of the proposed BREEN framework.
• Figure 4: Comparison on the performance when the amount of pretraining data increases from 4M to 9M
• Figure 5: Visualization of attention scores from output tokens to image tokens and learnable query tokens.