Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Gated Relational Alignment via Confidence-based Distillation for Efficient VLMs

Yanlong Chen, Amirhossein Habibian, Luca Benini, Yawei Li

TL;DR

This work tackles the cost of Vision-Language Models by proposing GRACE, an information-theoretic framework that jointly optimizes quantization-aware training and knowledge distillation. By treating the teacher as a dense source of task-relevant information, it introduces confidence-gated decoupled KD, Relational Centered Kernel Alignment, and an adaptive Information Bottleneck controller to allocate limited bit-budget capacity effectively. The approach leverages group-wise learned step-size quantization to enable real INT4 deployment, while the IB-based controller automatically balances fidelity to teacher knowledge against capacity constraints. Empirical results on LLaVA-1.5 and Qwen2-VL demonstrate that INT4 GRACE can surpass BF16 baselines and closely approach or even exceed teacher performance, with substantial throughput and memory benefits for edge deployment. Overall, GRACE provides a principled, practical solution for efficient multimodal inference at very low precision.

Abstract

Vision-Language Models (VLMs) achieve strong multimodal performance but are costly to deploy, and post-training quantization often causes significant accuracy loss. Despite its potential, quantization-aware training for VLMs remains underexplored. We propose GRACE, a framework unifying knowledge distillation and QAT under the Information Bottleneck principle: quantization constrains information capacity while distillation guides what to preserve within this budget. Treating the teacher as a proxy for task-relevant information, we introduce confidence-gated decoupled distillation to filter unreliable supervision, relational centered kernel alignment to transfer visual token structures, and an adaptive controller via Lagrangian relaxation to balance fidelity against capacity constraints. Across extensive benchmarks on LLaVA and Qwen families, our INT4 models consistently outperform FP16 baselines (e.g., LLaVA-1.5-7B: 70.1 vs. 66.8 on SQA; Qwen2-VL-2B: 76.9 vs. 72.6 on MMBench), nearly matching teacher performance. Using real INT4 kernel, we achieve 3$\times$ throughput with 54% memory reduction. This principled framework significantly outperforms existing quantization methods, making GRACE a compelling solution for resource-constrained deployment.

Gated Relational Alignment via Confidence-based Distillation for Efficient VLMs

TL;DR

This work tackles the cost of Vision-Language Models by proposing GRACE, an information-theoretic framework that jointly optimizes quantization-aware training and knowledge distillation. By treating the teacher as a dense source of task-relevant information, it introduces confidence-gated decoupled KD, Relational Centered Kernel Alignment, and an adaptive Information Bottleneck controller to allocate limited bit-budget capacity effectively. The approach leverages group-wise learned step-size quantization to enable real INT4 deployment, while the IB-based controller automatically balances fidelity to teacher knowledge against capacity constraints. Empirical results on LLaVA-1.5 and Qwen2-VL demonstrate that INT4 GRACE can surpass BF16 baselines and closely approach or even exceed teacher performance, with substantial throughput and memory benefits for edge deployment. Overall, GRACE provides a principled, practical solution for efficient multimodal inference at very low precision.

Abstract

Vision-Language Models (VLMs) achieve strong multimodal performance but are costly to deploy, and post-training quantization often causes significant accuracy loss. Despite its potential, quantization-aware training for VLMs remains underexplored. We propose GRACE, a framework unifying knowledge distillation and QAT under the Information Bottleneck principle: quantization constrains information capacity while distillation guides what to preserve within this budget. Treating the teacher as a proxy for task-relevant information, we introduce confidence-gated decoupled distillation to filter unreliable supervision, relational centered kernel alignment to transfer visual token structures, and an adaptive controller via Lagrangian relaxation to balance fidelity against capacity constraints. Across extensive benchmarks on LLaVA and Qwen families, our INT4 models consistently outperform FP16 baselines (e.g., LLaVA-1.5-7B: 70.1 vs. 66.8 on SQA; Qwen2-VL-2B: 76.9 vs. 72.6 on MMBench), nearly matching teacher performance. Using real INT4 kernel, we achieve 3 throughput with 54% memory reduction. This principled framework significantly outperforms existing quantization methods, making GRACE a compelling solution for resource-constrained deployment.
Paper Structure (62 sections, 5 theorems, 45 equations, 18 figures, 8 tables)

This paper contains 62 sections, 5 theorems, 45 equations, 18 figures, 8 tables.

Table of Contents

  1. Introduction
  2. Motivation
  3. Teacher Confidence and Supervision Quality
  4. Visual Attention Mismatch Across Model Scales
  5. Method
  6. Overview
  7. Confidence-Gated Decoupled Knowledge Distillation
  8. Relational Centered Kernel Alignment
  9. Group-wise Learned Step-size Quantization
  10. Adaptive IB Controller
  11. Experiments
  12. Main Results
  13. Ablation Studies
  14. Conclusion
  15. Additional Experiments
  16. ...and 47 more sections

Key Result

Theorem 3.1

Let $w_i = g_i / \sum_j g_j$ denote the normalized weights. The gated loss satisfies: where $\bar{\mathcal{L}}_{\text{DKD}} = \frac{1}{N}\sum_i \mathcal{L}_{\text{DKD}}^{(i)}$ is the unweighted average. Since $w_i$ decreases monotonically with $\tilde{h}_i$, positive correlation between entropy and loss implies $\mathrm{Cov}(w_i, \mathcal{L}_{\text{DKD}}^{(i)}) < 0$.

Figures (18)

  • Figure 1: Normalized performance comparison of advanced INT4 quantization methods on Qwen2-VL-2B. GRACE outperforms existing methods, including RTN, AWQ, SPEED-Q, GPTQ, MBQ.
  • Figure 2: Correlation between teacher entropy and error rate on ScienceQA (LLaVA-1.5 13B). (a) The scatter plot shows a linear relationship between entropy (x-axis) and error rate (y-axis), with a strong correlation. (b) The histogram illustrates the distribution of entropy values, with error rates overlaid to show how error increases with higher entropy.
  • Figure 3: Multi-layer attention visualization of LLaVA-1.5 13B (top) and 7B (bottom). Given the question "What object is being used as the telephone receiver?", the 13B model progressively localizes the banana across layers, while the 7B model exhibits scattered attention throughout.
  • Figure 4: Overview of the GRACE framework. A frozen LLaVA-1.5 13B teacher and a quantization-aware 7B student jointly process each input. The student receives three complementary supervisory signals: (i) Confidence-Gated DKD, which decomposes distillation into target-class and non-target-class components, weighting each token by teacher confidence to suppress noisy supervision; (ii) Relational CKA, which aligns centered kernel matrices $K_T$ and $K_S$ of visual tokens at the penultimate LLM layer, transferring relational structure (text tokens excluded); and (iii) an Adaptive IB Controller that monitors the EMA-smoothed gated loss $\widehat{\mathcal{L}}_{\text{GDKD}}$ and dynamically adjusts distillation strength $\beta$. Model weights $W$ and per-group quantization scales $s$ are updated jointly throughout training.
  • Figure 5: Visualization of pairwise visual token similarity from LLaVA-1.5 13B. The heatmap shows cosine similarity between token 0 (yellow box, located in the sky region) and all other tokens in the spatial grid, where Patch X and Patch Y denote the horizontal and vertical grid coordinates respectively.
  • ...and 13 more figures

Theorems & Definitions (8)

  • Theorem 3.1: Effect of Confidence Gating
  • Proposition 3.2: Variational Lower Bound and KL Gap
  • Theorem 2.1: Fano's Inequality
  • Theorem : Restated
  • proof
  • Proposition : Restated
  • proof
  • Remark 2.5: Tightness of the Bound