Instruction Tuning-free Visual Token Complement for Multimodal LLMs

TL;DR

This work targets the visual information loss in multimodal LLMs caused by caption- or instruction-driven VPGs. It introduces Visual Token Complement (VTC), a reconstruction-prior framework that learns complementary visual tokens $\boldsymbol{y}'$ via a learned function $\boldsymbol{c}=\text{VTC}(\boldsymbol{h},\boldsymbol{y})$ and forms $\hat{\boldsymbol{y}}=\text{Concat}(\boldsymbol{y},\boldsymbol{y}')$, enabling a richer, multi-level vision representation. Training is unsupervised, using a frozen image encoder, VPG, and Stable Diffusion as the decoder, with a projection matrix $\mathbf{W}$ aligning LLM and CLIP embeddings through $\boldsymbol{v}_{CLIP} \approx \boldsymbol{v}_{LLM}\mathbf{W}$ under $\mathbf{W}^T\mathbf{W}=\mathbf{I}$; at inference, VTC supports iterative refinement $\boldsymbol{y}^{k+1}=\text{VPG}(\boldsymbol{c}^k,\boldsymbol{q})$, enabling on-demand completion. Empirically, VTC improves zero-shot performance across LVLM-eHub, MME, and DEMON benchmarks, yields notable gains in visual dialog, and achieves competitive ChatGPT-4(V) evaluation scores, while decoupling vision-language learning from instruction tuning. These results demonstrate a practical, instruction-tuning-free pathway to enrich MLLMs with fuller visual semantics and interpretable token-level visual evidence.

Abstract

As the open community of large language models (LLMs) matures, multimodal LLMs (MLLMs) have promised an elegant bridge between vision and language. However, current research is inherently constrained by challenges such as the need for high-quality instruction pairs and the loss of visual information in image-to-text training objectives. To this end, we propose a Visual Token Complement framework (VTC) that helps MLLMs regain the missing visual features and thus improve response accuracy. Specifically, our VTC integrates text-to-image generation as a guide to identifying the text-irrelevant features, and a visual selector is then developed to generate complementary visual tokens to enrich the original visual input. Moreover, an iterative strategy is further designed to extract more visual information by iteratively using the visual selector without any additional training. Notably, the training pipeline requires no additional image-text pairs, resulting in a desired instruction tuning-free property. Both qualitative and quantitative experiments demonstrate the superiority and efficiency of our VTC.

Figures (11)

• Figure 1: Comparisons of our proposed Reconstruction prior (c) for training Visual Token Complement (VTC), Caption prior (a), and Instruction prior (b) for training Visual Prompt Generator (VPG). The attention maps of VPG and VTC are obtained by averaging the query-patch attention weights(Sec. \ref{['sec:vtc']}).
• Figure . 1: More visualizations of the complementary tokens in the first iteration (VTC(InstructBLIP)-1) and second iteration (VTC(InstructBLIP)-2). We also provide the retrieved tags and predicted captions.
• Figure 2: Iterative complement at the inference stage. Once trained, Our VTC can be reused multiple times to re-pick up the visual concepts omitted by the previous VPG. We showcase three times complements with increasingly detailed output captions.
• Figure . 2: Examples of the test dialogs.
• Figure 3: (a) Training pipeline of the proposed VTC, and (b) structure of VTC.