Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Fine-Grained Verifiers: Preference Modeling as Next-token Prediction in Vision-Language Alignment

Chenhang Cui, An Zhang, Yiyang Zhou, Zhaorun Chen, Gelei Deng, Huaxiu Yao, Tat-Seng Chua

TL;DR

This work tackles misalignment in Vision-Language Large Models by addressing hallucinations through fine-grained, token-level feedback from a model’s own vision encoder. It introduces FiSAO, a self-alignment framework that rewards individual generated tokens based on how well they align with visual input, thereby avoiding external data and costly annotations. The authors provide a theoretical justification showing vision-feedback can improve outputs, and demonstrate empirically that token-level rewards outperform sentence-level signals across multiple benchmarks, including LLaVA-1.5 and InstructBLIP backbones, with substantial performance gains over existing preference-tuning methods. Overall, FiSAO offers a data-efficient, scalable path to stronger vision-language alignment with practical benefits for safety and grounding in VLLMs.

Abstract

The recent advancements in large language models (LLMs) and pre-trained vision models have accelerated the development of vision-language large models (VLLMs), enhancing the interaction between visual and linguistic modalities. Despite their notable success across various domains, VLLMs face challenges in modality alignment, which can lead to issues like hallucinations and unsafe content generation. Current alignment techniques often rely on coarse feedback and external datasets, limiting scalability and performance. In this paper, we propose FiSAO (Fine-Grained Self-Alignment Optimization), a novel self-alignment method that utilizes the model's own visual encoder as a fine-grained verifier to improve vision-language alignment without the need for additional data. By leveraging token-level feedback from the vision encoder, FiSAO significantly improves vision-language alignment, even surpassing traditional preference tuning methods that require additional data. Through both theoretical analysis and experimental validation, we demonstrate that FiSAO effectively addresses the misalignment problem in VLLMs, marking the first instance of token-level rewards being applied to such models.

Fine-Grained Verifiers: Preference Modeling as Next-token Prediction in Vision-Language Alignment

TL;DR

This work tackles misalignment in Vision-Language Large Models by addressing hallucinations through fine-grained, token-level feedback from a model’s own vision encoder. It introduces FiSAO, a self-alignment framework that rewards individual generated tokens based on how well they align with visual input, thereby avoiding external data and costly annotations. The authors provide a theoretical justification showing vision-feedback can improve outputs, and demonstrate empirically that token-level rewards outperform sentence-level signals across multiple benchmarks, including LLaVA-1.5 and InstructBLIP backbones, with substantial performance gains over existing preference-tuning methods. Overall, FiSAO offers a data-efficient, scalable path to stronger vision-language alignment with practical benefits for safety and grounding in VLLMs.

Abstract

The recent advancements in large language models (LLMs) and pre-trained vision models have accelerated the development of vision-language large models (VLLMs), enhancing the interaction between visual and linguistic modalities. Despite their notable success across various domains, VLLMs face challenges in modality alignment, which can lead to issues like hallucinations and unsafe content generation. Current alignment techniques often rely on coarse feedback and external datasets, limiting scalability and performance. In this paper, we propose FiSAO (Fine-Grained Self-Alignment Optimization), a novel self-alignment method that utilizes the model's own visual encoder as a fine-grained verifier to improve vision-language alignment without the need for additional data. By leveraging token-level feedback from the vision encoder, FiSAO significantly improves vision-language alignment, even surpassing traditional preference tuning methods that require additional data. Through both theoretical analysis and experimental validation, we demonstrate that FiSAO effectively addresses the misalignment problem in VLLMs, marking the first instance of token-level rewards being applied to such models.

Paper Structure

This paper contains 35 sections, 1 theorem, 27 equations, 10 figures, 8 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Preliminaries
  3. FiSAO
  4. Empirical Findings
  5. Theoretical Framework for Incorporating Pre-trained Vision Models' Feedback into Model Training
  6. Reward Modeling for FiSAO
  7. Generation from the Perspective of Sequential Decision-Making
  8. Estimation of Baseline Scores for Ground Truth and Hallucinated Distributions
  9. fine-grained Reward Calculation
  10. Fine-Grained Preference Policy Optimization for FiSAO
  11. Experiment
  12. Experimental Setup
  13. Experimental Results on Benckmarks (RQ1)
  14. Analysis (RQ2&RQ3)
  15. Related Work
  16. ...and 20 more sections

Key Result

Theorem 3.1

Suppose that $\pi_{\theta_{t}}^*(y \mid x)$ lies in the LLM space $\{ \pi_{\theta}(y \mid x): \theta \in \Theta \}$. Then, there exists some $\lambda$ > 0 , such that $\mathbb{E}_{\pi_{\theta (\lambda)}(y \mid x)}[L(y)] < \mathbb{E}_{\pi_{\theta (0)}(y \mid x)}[L(y)].$

Figures (10)

  • Figure 1: Comparison of token-level (\ref{['fig:image1_re']}) and sentence-level (\ref{['fig:image2_re']}) reward distributions for hallucinated and correct objects in the LLaVA 1.5 model. Further comparisons can be found in Appendix \ref{['sec:add_exp']}.
  • Figure 2: Correlation between the CLIP-based sentence rewards and conventional evaluation metrics: BLEU (\ref{['fig:image1_cor']}) and ROUGE (\ref{['fig:image2_cor']}). A small Pearson correlation coefficient ($r$) indicates a weak correlation. More comparison is detailed in Appendix \ref{['sec:add_exp']}.
  • Figure 3: The overall framework of FiSAO. We employ two steps to achieve self-alignment from fine-grained feedback: (1) calculate the fine-grained reward based on the baseline score obtained from correct and hallucinated tokens. (2) optimize the preference policy using this reward to align the model’s responses during training.
  • Figure 4: Comparison of reward distributions for generated objects on LLaVA-1.5 before and after Training.
  • Figure 5: Case study on sentence-level reward and token-level reward.
  • ...and 5 more figures

Theorems & Definitions (1)

  • Theorem 3.1