GenRecal: Generation after Recalibration from Large to Small Vision-Language Models

TL;DR

GenRecal addresses the challenge of distilling knowledge from large vision-language models to smaller ones when the teacher and student use different token types. It introduces a Recalibrator that learns to map small-VLM features into the large-VLM latent space via an autoregressive loss and a $D_{KL}$ regularization, enabling token-types-agnostic distillation. Empirically, GenRecal improves over traditional distillation and SFT across diverse benchmarks and model pairs, with larger gains when both teacher and student are stronger, and it remains efficient by removing the Recalibrator at inference. This framework broadens the applicability of on-device VLM deployment and paves the way for flexible, cross-architecture knowledge transfer in multimodal models.

Abstract

Recent advancements in vision-language models (VLMs) have leveraged large language models (LLMs) to achieve performance on par with closed-source systems like GPT-4V. However, deploying these models in real-world scenarios, particularly on resource-constrained devices, remains challenging due to their substantial computational demands. This has spurred interest in distilling knowledge from large VLMs into smaller, more efficient counterparts. A key challenge arises here from the diversity of VLM architectures, which are built on different LLMs and employ varying token types-differing in vocabulary size, token splits, and token index ordering. To address this challenge of limitation to a specific VLM type, we present Generation after Recalibration (GenRecal), a general-purpose distillation framework for VLMs. GenRecal incorporates a Recalibrator that aligns and adapts feature representations between heterogeneous VLMs, enabling effective knowledge transfer across different types of VLMs. Through extensive experiments on multiple challenging benchmarks, we demonstrate that GenRecal significantly improves baseline performances, eventually outperforming large-scale open- and closed-source VLMs.

Figures (11)

• Figure 1: (Left) Visualizing the token indices of a given image and text prompt and representing the possibility of distillation among various VLM pair combinations, comparing traditional distillation with our proposed distillation framework, GenRecal. Note that the parentheses mean each VLM's LLM tokenizer, '...' indicates the placement of image features, and the number of these features varies depending on the image embedding strategy. (Right) Comparing the performance of a challenging evaluation benchmark, MM-Vet yu2023mm, with [A] baseline, [B] SFT on the baseline, [C] traditional distillation and [D] GenRecal from same token types of large VLMs, and GenRecal with more powerful [E] large and [F] small VLMs.
• Figure 2: (Left) Comparison of the challenging benchmark performances, MMB liu2023mmbench, MM-Vet yu2023mm, MMMU yue2023mmmu, and MMMU-Pro yue2024mmmu by changing large VLMs. The more powerful large VLMs we select, the greater the performance improvement we can achieve. (Right) Comparing the performance of the challenging benchmark: MMMU yue2023mmmu, with GenRecal and various VLMs across model sizes. Note that all the experiments in \ref{['fig:1']} and \ref{['fig:2']} are conducted on the equal training dataset.
• Figure 3: Overview of the GenRecal architecture and its training stages. We denote the question and answer tokens from the small VLM as $q_s$ and $a_s$, and those from the large VLM as $q_l$ and $a_l$. For simplicity, the vision encoder and image-token projector are omitted from the illustration. Note that Recalibrator is used only during training as a bridge between heterogeneous VLMs. During inference, Recalibrator is removed, leaving the small VLM’s architecture unchanged and adding no extra computational cost.
• Figure 4: Overview of regularization simultaneously done with first training stage-alignment.
• Figure 5: Distillation performance on MMMU yue2023mmmu for various pairings os teacher and student VLM. Each cell indicates the resulting score when using the corresponding teacher (rows) and student (columns) model sizes.