Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

LinguDistill: Recovering Linguistic Ability in Vision- Language Models via Selective Cross-Modal Distillation

Patrick Amadeus Irawan, Erland Hilman Fuadi, Shanu Kumar, Alham Fikri Aji, Yova Kementchedjhieva

Abstract

Adapting pretrained language models (LMs) into vision-language models (VLMs) can degrade their native linguistic capability due to representation shift and cross-modal interference introduced during multimodal adaptation. Such loss is difficult to recover, even with targeted task-specific fine-tuning using standard objectives. Prior recovery approaches typically introduce additional modules that act as intermediate alignment layers to maintain or isolate modality-specific subspaces, which increases architectural complexity, adds parameters at inference time, and limits flexibility across models and settings. We propose LinguDistill, an adapter-free distillation method that restores linguistic capability by utilizing the original frozen LM as a teacher. We overcome the key challenge of enabling vision-conditioned teacher supervision by introducing layer-wise KV-cache sharing, which exposes the teacher to the student's multimodal representations without modifying the architecture of either model. We then selectively distill the teacher's strong linguistic signal on language-intensive data to recover language capability, while preserving the student's visual grounding on multimodal tasks. As a result, LinguDistill recovers $\sim$10% of the performance lost on language and knowledge benchmarks, while maintaining comparable performance on vision-heavy tasks. Our findings demonstrate that linguistic capability can be recovered without additional modules, providing an efficient and practical solution to modality-specific degradation in multimodal models.

LinguDistill: Recovering Linguistic Ability in Vision- Language Models via Selective Cross-Modal Distillation

Abstract

Adapting pretrained language models (LMs) into vision-language models (VLMs) can degrade their native linguistic capability due to representation shift and cross-modal interference introduced during multimodal adaptation. Such loss is difficult to recover, even with targeted task-specific fine-tuning using standard objectives. Prior recovery approaches typically introduce additional modules that act as intermediate alignment layers to maintain or isolate modality-specific subspaces, which increases architectural complexity, adds parameters at inference time, and limits flexibility across models and settings. We propose LinguDistill, an adapter-free distillation method that restores linguistic capability by utilizing the original frozen LM as a teacher. We overcome the key challenge of enabling vision-conditioned teacher supervision by introducing layer-wise KV-cache sharing, which exposes the teacher to the student's multimodal representations without modifying the architecture of either model. We then selectively distill the teacher's strong linguistic signal on language-intensive data to recover language capability, while preserving the student's visual grounding on multimodal tasks. As a result, LinguDistill recovers 10% of the performance lost on language and knowledge benchmarks, while maintaining comparable performance on vision-heavy tasks. Our findings demonstrate that linguistic capability can be recovered without additional modules, providing an efficient and practical solution to modality-specific degradation in multimodal models.

Paper Structure

This paper contains 47 sections, 3 equations, 2 figures, 10 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Preserving Language Capability in Multimodal Models.
  4. Cross-Modal Knowledge Distillation.
  5. KV-Cache Sharing across Model Components.
  6. LinguDistill
  7. KV Sharing Architecture
  8. Selective Distillation Objective.
  9. Setup
  10. Dataset Design
  11. Models & Optimization
  12. Experiment Runs
  13. Evaluation
  14. Results
  15. Standard fine-tuning degrades language capability.
  16. ...and 32 more sections

Figures (2)

  • Figure 1: Training-loss analysis for the three selective distillation variants. Left: the CE term is lowest for the main LinguDistill setting, with High KD slightly above it and Low KD clearly worse. Right: the Soft loss uses the combined logged objective, which mixes the teacher-weighted distillation signal with the VLM loss in a balanced manner.
  • Figure 2: Pseudo-code for LinguDistill . The image is first mapped into the language subspace, the left decoder absorbs the full multimodal prompt and materializes prompt KV memory, and the frozen right decoder continues from that inherited state to produce detached supervision. The optimization objective combines source-selective distillation with hard next-token supervision, while updating only the left-side parameters.