HaploVL: A Single-Transformer Baseline for Multi-Modal Understanding

TL;DR

HaploVL introduces a single-transformer multi-modal baseline that fuses vision and text at an early stage to enable end-to-end, autoregressive multi-modal generation with reduced data and compute. It combines lightweight multi-modal embeddings, a pre-decoder that distills vision cues from a fixed teacher (CLIP-ViT-L), and a post-decoder that leverages LLM knowledge for generation, trained via a two-stage process that preserves vision capabilities while learning new multi-modal skills. The approach yields competitive or superior performance relative to existing unified LMMs and narrows the gap with compositional models, particularly in fine-grained perception tasks, while reducing resource requirements. This work provides a practical and scalable baseline for single-transformer multi-modal models and offers insights into effective pre-training and fine-tuning strategies that leverage pre-trained components.

Abstract

Recent advancements in large language models (LLMs) have significantly propelled the development of large multi-modal models (LMMs), highlighting the potential for general and intelligent assistants. However, most LMMs model visual and textual modalities separately, leading to recent efforts to develop native LMMs using a single transformer. Despite the promise, these native models are resource-intensive and often exhibit performance gaps compared to their compositional counterparts. To alleviate this issue, we propose a simple yet efficient method to construct a baseline for the native and end-to-end large multi-modal model in a single transformer. First, we propose a new early-fusion LMM that can fuse multi-modal inputs in the early stage and respond to visual instructions in an auto-regressive manner. Second, we devise an efficient training recipe for the proposed model, which harnesses the prior knowledge of the pre-trained models, addressing both the performance limitations and the challenge of resource consumption. The proposed model demonstrates superior performance compared to other LMMs using one transformer and significantly narrows the performance gap with compositional LMMs.

Figures (6)

• Figure 1: Performance comparison with single-transformer models on multi-modal understanding benchmarks. Our HaploVL demonstrates superiority over other counterparts.
• Figure 2: Architecture comparison with the compositional LMM llava_v1_5, EVE eve. In our HaploVL, the pre-decoder dynamically extracts vision cues based on the input text, and the post-decoder further fuses the multi-modal embeddings. Our model inherits the prior knowledge from vision and language models, thus requiring less data than EVE.
• Figure 3: The diagram of HaploVL. It includes a transformer decoder made up of a pre-decoder and a post-decoder. During the pre-training stage (a), the pre-decoder is trained by distilling knowledge from the pre-trained vision encoder and the text embeddings of the LLM. Heads and teacher models are dropped after pre-training. In the full fine-tuning stage (b), the entire model is fine-tuned using visual instruction data.
• Figure 4: Qualitative comparison of LLaVA-1.5-7B llava_v1_5 and our HaploVL-7B. The first line involves cases about fine-grained perception. The second line includes cases of logical reasoning that depend on fine-grained perception.
• Figure 5: Visualization for the early fusion mechanism of our single transformer. The second row illustrates the attention map of the gray words concerning the vision embeddings after the pre-decoder.