Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Online In-Context Distillation for Low-Resource Vision Language Models

Zhiqi Kang, Rahaf Aljundi, Vaggelis Dorovatas, Karteek Alahari

TL;DR

This work tackles the challenge of adapting vision-language models to low-resource settings by introducing Online In-Context Distillation (ICD), a training-free framework where a small student model inherits knowledge from a large teacher through inference-time demonstrations. ICD integrates test-time scaling to improve teacher annotations, a cross-modal demonstration selection mechanism to robustly curate demonstrations, and uncertainty-aware querying to limit teacher calls while ensuring informative context. The approach is analyzed to identify when ICL can benefit small VLMs (notably in the 4B–12B range) and demonstrates that training-free ICD can bridge much of the gap to larger teachers with modest compute and annotation budgets, achieving average gains around 14.8% across tasks and limited teacher annotation rates (~14.7%). Overall, ICD offers a practical, scalable path to deploy capable VLMs on edge devices by avoiding fine-tuning and heavy annotation while maintaining strong performance relative to zero-shot and, in some cases, the teacher’s zero-shot capabilities.

Abstract

As the field continues its push for ever more resources, this work turns the spotlight on a critical question: how can vision-language models (VLMs) be adapted to thrive in low-resource, budget-constrained settings? While large VLMs offer strong performance, they are impractical to deploy in such settings. Small VLMs, on the other hand, are efficient but typically require costly fine-tuning to close the performance gap with larger models in the deployment domain. Inspired by the in-context learning framework, we propose an online In-Context Distillation (ICD) method, in which a small VLM collaborates with a stronger teacher model at inference time, distilling its knowledge via sparse demonstrations to efficiently bridge the gap between them. Our method is built on an in-depth analysis that identifies the scale and the choice of models for which vision-language ICL is currently feasible, and demonstrates the advantage of ICL over fine-tuning under constrained compute budgets. We enhance our method with a novel cross-modal demonstration selection strategy, teacher test-time scaling to reduce noise, and student uncertainty conditioning to dynamically populate a demonstration pool and minimize teacher queries. Our ICD method significantly boosts the performance of small models (up to 33%) using scarce teacher annotations (as low as 4%), and competes with the teacher's zero-shot performance.

Online In-Context Distillation for Low-Resource Vision Language Models

TL;DR

This work tackles the challenge of adapting vision-language models to low-resource settings by introducing Online In-Context Distillation (ICD), a training-free framework where a small student model inherits knowledge from a large teacher through inference-time demonstrations. ICD integrates test-time scaling to improve teacher annotations, a cross-modal demonstration selection mechanism to robustly curate demonstrations, and uncertainty-aware querying to limit teacher calls while ensuring informative context. The approach is analyzed to identify when ICL can benefit small VLMs (notably in the 4B–12B range) and demonstrates that training-free ICD can bridge much of the gap to larger teachers with modest compute and annotation budgets, achieving average gains around 14.8% across tasks and limited teacher annotation rates (~14.7%). Overall, ICD offers a practical, scalable path to deploy capable VLMs on edge devices by avoiding fine-tuning and heavy annotation while maintaining strong performance relative to zero-shot and, in some cases, the teacher’s zero-shot capabilities.

Abstract

As the field continues its push for ever more resources, this work turns the spotlight on a critical question: how can vision-language models (VLMs) be adapted to thrive in low-resource, budget-constrained settings? While large VLMs offer strong performance, they are impractical to deploy in such settings. Small VLMs, on the other hand, are efficient but typically require costly fine-tuning to close the performance gap with larger models in the deployment domain. Inspired by the in-context learning framework, we propose an online In-Context Distillation (ICD) method, in which a small VLM collaborates with a stronger teacher model at inference time, distilling its knowledge via sparse demonstrations to efficiently bridge the gap between them. Our method is built on an in-depth analysis that identifies the scale and the choice of models for which vision-language ICL is currently feasible, and demonstrates the advantage of ICL over fine-tuning under constrained compute budgets. We enhance our method with a novel cross-modal demonstration selection strategy, teacher test-time scaling to reduce noise, and student uncertainty conditioning to dynamically populate a demonstration pool and minimize teacher queries. Our ICD method significantly boosts the performance of small models (up to 33%) using scarce teacher annotations (as low as 4%), and competes with the teacher's zero-shot performance.
Paper Structure (64 sections, 12 equations, 17 figures, 10 tables, 4 algorithms)

This paper contains 64 sections, 12 equations, 17 figures, 10 tables, 4 algorithms.

Table of Contents

  1. Introduction
  2. Related Work
  3. In-context learning.
  4. In-context learning for VLMs.
  5. VLMs distillation.
  6. Can we apply ICL to small VLMs?
  7. Definition of In-Context Learning
  8. A Feasibility Test of Small VLMs
  9. Vision Language ICL (VL ICL) Failure Modes: A Case Study.
  10. ICL potential for low-resource adaptation.
  11. ICD: In-Context Distillation
  12. Improving Annotations with Test-Time Scaling
  13. Improving Demonstration Selection with Cross-Modal Matching
  14. Uncertainty-Aware ICL for Online Evaluation
  15. Experiments
  16. ...and 49 more sections

Figures (17)

  • Figure 1: Left: Overview of our online In-Context Distillation framework. The teacher model in the cloud receives demonstration requests from the student and provides refined answers with test-time scaling. The small student model accesses the dynamically populated pool for answering queries and upon uncertainty, it queries the teacher for new demonstrations. Right: Performance vs. compute budget normalized by that of GPT-4o. Our ICD boosts small VLMs toward the performance of large teacher models with moderate compute overhead.
  • Figure 2: ICL evaluation for different InternVL sizes and versions. Left to right: tasks are ordered with increasing difficulty. Within the same version (i.e., version 2.5), small VLMs exhibit better ICL capability than tiny VLMs, though tiny VLMs (i.e., size 4B) also improve across versions.
  • Figure 3: ICL evaluation to verify models' inherent capabilities (best viewed in color). From left to right: CUB (Classification): image label prediction. CUB (Multi options): (A,B,.. prediction). Language ICL: converted text-only ICL. Image OCR: text recognition from the image. Induce operator: inducing the operator represented by the question mark. Calculate results: final evaluation.
  • Figure 4: Comparison of offline fine-tuning v.s. ICL under a compute budget. ICL is a more appealing choice in a low-source environment as it brings significant performance improvement at low computational cost.
  • Figure 5: Accumulative accuracy vs. compute cost in an online scenario. Our ICD excels at this tradeoff.
  • ...and 12 more figures