VLsI: Verbalized Layers-to-Interactions from Large to Small Vision Language Models

TL;DR

VLsI addresses the deployment- and efficiency-related bottlenecks of vision-language models by introducing layer-wise verbalized distillation from a large backbone to smaller 2B/7B backbones. It couples verbalizers that map intermediate features into natural language space with an adaptive layer-matching strategy based on KL-divergence to align the small model's reasoning progression with the large model, followed by supervised finetuning. The approach yields significant improvements over GPT-4V across ten benchmarks (11.0% for 2B and 17.4% for 7B) without architectural changes, scaling, or merging, demonstrating the practicality of language as a medium for cross-model knowledge transfer. This work offers a scalable pathway for deploying capable VLMs on resource-constrained devices, leveraging natural language to facilitate precise, layer-wise knowledge transfer between models of different scales.

Abstract

The recent surge in high-quality visual instruction tuning samples from closed-source vision-language models (VLMs) such as GPT-4V has accelerated the release of open-source VLMs across various model sizes. However, scaling VLMs to improve performance using larger models brings significant computational challenges, especially for deployment on resource-constrained devices like mobile platforms and robots. To address this, we propose VLsI: Verbalized Layers-to-Interactions, a new VLM family in 2B and 7B model sizes, which prioritizes efficiency without compromising accuracy. VLsI leverages a unique, layer-wise distillation process, introducing intermediate "verbalizers" that map features from each layer to natural language space, allowing smaller VLMs to flexibly align with the reasoning processes of larger VLMs. This approach mitigates the training instability often encountered in output imitation and goes beyond typical final-layer tuning by aligning the small VLMs' layer-wise progression with that of the large ones. We validate VLsI across ten challenging vision-language benchmarks, achieving notable performance gains (11.0% for 2B and 17.4% for 7B) over GPT-4V without the need for model scaling, merging, or architectural changes.

Figures (27)

• Figure 1: Performance overview of VLsI on vision-language benchmarks. (a) Accuracy on MM-Vet yu2023mm for various model sizes, showing that VLsI (2B and 7B) achieves competitive performance compared to proprietary closed-source VLMs. (b) Comparative evaluation on multiple challenging benchmarks, where VLsI (green and blue) outperforms leading closed-source VLMs, including GPT-4V gpttechnical, Claude-3.5-Sonnet claude3series2024, and Gemini-1.5-Pro team2023gemini, highlighting its efficiency and effectiveness across diverse tasks.
• Figure 3: Example of verbalized outputs from each intermediate target layer in an alternative small-backbone VLM (without VLsI enhancements) and the VLsI. The visual question prompts VLM to predict the missing image in a sequence pattern. The outputs illustrate how each layer progressively interprets the visual cues, with VLsI accurately identifying the answer as 'a star with a dot' in the final layer, while the alternative small-backbone VLM incorrectly predicts 'a diamond with a dot'. This demonstrates the improved interpretative capability of VLsI through layer-wise, language-based distillation.
• Figure 4: Comparison of performance on MM-Vet yu2023mm and MMMU yue2023mmmu across different model size combinations in large and small backbone VLMs. Each cell shows the evaluation results for various interaction configurations between 0.5B, 2B, and 7B small backbone VLMs trained with either Qwen2-VL wang2024qwen2vl or LLaVA-OV li2024llava as the large-backbone VLM.
• Figure 5: Distribution changes of the matched indices between small-backbone and large-backbone VLMs at the interaction step. The left figure shows the distribution at the beginning of training, while the right figure shows it at the end.
• Figure : (a) Verbalization Step