Token-Level Inference-Time Alignment for Vision-Language Models

TL;DR

Vision-Language Models often hallucinate due to reliance on language priors when visual grounding is weak. The paper proposes TITA, a token-level inference-time alignment framework that freezes the VLM and trains a lightweight autoregressive reward model; during generation, token-level log-probability ratios guide decoding without fine-tuning the backbone. TITA aggregates diverse augmented views to form informative preference data, and a logit-mapping mechanism ensures compatibility across tokenizers, enabling plug-and-play deployment. The approach yields consistent improvements across 12 benchmarks and multiple LVLMs with minimal overhead, offering a practical, scalable path to more trustworthy multimodal reasoning.

Abstract

Vision-Language Models (VLMs) have become essential backbones of modern multimodal intelligence, yet their outputs remain prone to hallucination-plausible text misaligned with visual inputs. Existing alignment approaches often rely on expensive fine-tuning with annotated preference data or sequence-level inference strategies that provide only coarse, delayed feedback. To overcome these limitations, we present TITA (Token-level Inference-Time Alignment), a lightweight framework that freezes the base VLM and instead trains a reward model to approximate its distribution. During inference, implicit preference signals are extracted as log-probability ratios between the reward model and the target VLM, yielding dense autoregressive feedback. This formulation can be viewed as an inference-time variant of Direct Preference Optimization (DPO), providing token-level corrective signals without retraining the backbone. Extensive evaluations on LLaVA-1.5-7B and 13B show consistent gains across 12 benchmarks, with improvements of 8.6% on MMVet and 6.7% on POPE, indicating stronger general understanding and reduced hallucinations. Additional experiments on Qwen2.5-VL-7B and DeepSeek-VL2-27.5B show comparable gains, especially in hallucination reduction and VQA accuracy, while incurring negligible inference overhead.

Figures (6)

• Figure 1: Overview of preference alignment strategies for VLMs (LLaVA-1.5-7B). (A) Training-time alignment fine-tunes base model $\pi_{\theta}$ with human-labeled preferences. (B) Sequence-level inference-time alignment reranks complete responses with reward models. (C) TITA with token-level decoding guidance via implicit preference optimization for lightweight and fine-grained alignment.
• Figure 2: Attention visualization demonstrating how TITA enables holistic caption generation. The winner answer $y_\text{w}$ is generated by fusing multiple responses obtained from augmented versions of the image, capturing more comprehensive and details compared to the original generation $y_\text{l}$.
• Figure 3: Ablation studies on reward integration and reward modeling: (a) MMVet accuracy under different scale factor $\lambda$ in Equation \ref{['eq:output']}. TITA achieves optimal performance at $\lambda = 0.6$. (b) Comparison of single-view versus fusion-based reward modeling. Fusion-based multi-view preference construction consistently improves performance across MMVet, MMB, and POPE benchmarks.
• Figure 4: Visualization of response-token attention over visual features on the POPE benchmark. Compared to the baseline LLaVA-1.5-7B, TITA-guided inference produces higher and more focused attention weights on visually grounded tokens.
• Figure 5: Comparison of 12 data augmentation strategies applied to LLaVA-1.5, including various geometric and color transformations as well as contrast learning enhancement methods. By analyzing these methods, the goal is to find the combination that best improves the performance of LVLMs.