Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Delta-LLaVA: Base-then-Specialize Alignment for Token-Efficient Vision-Language Models

Mohamad Zamini, Diksha Shukla

TL;DR

Delta-LLaVA introduces a token-efficient visual projector built around a low-rank DeltaProjection for base alignment, followed by lightweight specialization blocks to handle global and local structure under strict token budgets. The base-then-specialize design reduces visual token counts dramatically while preserving, and often improving, reasoning and grounding across VQA, captioning, and specialized tasks. Efficiency analyses and extensive ablations show substantial throughput gains and faster training, without sacrificing downstream performance, especially when scaling to larger LLM backbones. The approach highlights the importance of principled token formation prior to interaction capacity scaling for scalable, real-time multimodal systems.

Abstract

Multimodal Large Language Models (MLLMs) combine visual and textual representations to enable rich reasoning capabilities. However, the high computational cost of processing dense visual tokens remains a major bottleneck. A critical component in this pipeline is the visual projector, which bridges the vision encoder and the language model. Standard designs often employ a simple multi-layer perceptron for direct token mapping, but this approach scales poorly with high-resolution inputs, introducing significant redundancy. We present Delta-LLaVA, a token-efficient projector that employs a low-rank DeltaProjection to align multi-level vision features into a compact subspace before further interaction. On top of this base alignment, lightweight Transformer blocks act as specialization layers, capturing both global and local structure under constrained token budgets. Extensive experiments and ablations demonstrate that this base-then-specialize design yields consistent gains across multiple benchmarks with only 144 tokens, highlighting the importance of token formation prior to scaling interaction capacity. With Delta-LLaVA, inference throughput improves by up to 55%, while end-to-end training accelerates by nearly 4-5x in pretraining and over 1.5x in finetuning, highlighting the dual benefits of our design in both efficiency and scalability.

Delta-LLaVA: Base-then-Specialize Alignment for Token-Efficient Vision-Language Models

TL;DR

Delta-LLaVA introduces a token-efficient visual projector built around a low-rank DeltaProjection for base alignment, followed by lightweight specialization blocks to handle global and local structure under strict token budgets. The base-then-specialize design reduces visual token counts dramatically while preserving, and often improving, reasoning and grounding across VQA, captioning, and specialized tasks. Efficiency analyses and extensive ablations show substantial throughput gains and faster training, without sacrificing downstream performance, especially when scaling to larger LLM backbones. The approach highlights the importance of principled token formation prior to interaction capacity scaling for scalable, real-time multimodal systems.

Abstract

Multimodal Large Language Models (MLLMs) combine visual and textual representations to enable rich reasoning capabilities. However, the high computational cost of processing dense visual tokens remains a major bottleneck. A critical component in this pipeline is the visual projector, which bridges the vision encoder and the language model. Standard designs often employ a simple multi-layer perceptron for direct token mapping, but this approach scales poorly with high-resolution inputs, introducing significant redundancy. We present Delta-LLaVA, a token-efficient projector that employs a low-rank DeltaProjection to align multi-level vision features into a compact subspace before further interaction. On top of this base alignment, lightweight Transformer blocks act as specialization layers, capturing both global and local structure under constrained token budgets. Extensive experiments and ablations demonstrate that this base-then-specialize design yields consistent gains across multiple benchmarks with only 144 tokens, highlighting the importance of token formation prior to scaling interaction capacity. With Delta-LLaVA, inference throughput improves by up to 55%, while end-to-end training accelerates by nearly 4-5x in pretraining and over 1.5x in finetuning, highlighting the dual benefits of our design in both efficiency and scalability.

Paper Structure

This paper contains 21 sections, 12 equations, 3 figures, 5 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Method
  4. Delta-LLaVA: Multimodal Projector
  5. Query construction.
  6. DeltaProjection.
  7. Coarse-to-fine mixing.
  8. Feed-forward refinement.
  9. FLOPs Analysis.
  10. Projector.
  11. LLM Prefill.
  12. LLM Decode.
  13. Vision Encoder.
  14. Experiments
  15. Empirical Analysis.
  16. ...and 6 more sections

Figures (3)

  • Figure 1: Grad-CAM visualization of the prompt: What is weird about this picture? The response (144 tokens) is: The weird aspect of this picture is that a man is standing on the back of a yellow taxi while holding a clothes iron. This is unusual because it is not common to see someone using a clothes iron while standing on the back of a moving vehicle, especially a taxi. The man's actions seem unconventional and potentially dangerous, as he could lose his balance and fall off the taxi, causing injury or damage to the vehicle. The images show results using 16, 64, and 144 visual tokens, from left to right (after the original image).
  • Figure 3: Inference performance and accuracy with naturally terminated generations ($\approx$30 new tokens on average) under varying visual-token counts.
  • Figure 4: Comparison of captioning robustness under token pruning. Across Flickr30k and Nocaps benchmarks, Delta-LLaVA achieves a superior accuracy--efficiency trade-off relative to FastV, especially at moderate to aggressive pruning levels.