Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Attention Guided Alignment in Efficient Vision-Language Models

Shweta Mahajan, Hoang Le, Hyojin Park, Farzad Farhadzadeh, Munawar Hayat, Fatih Porikli

TL;DR

The paper addresses object hallucination in efficient vision–language models by diagnosing weak visual grounding in concatenation-based architectures. It introduces AGE-VLM, which interleaves cross-attention in a decoder-only LLM and distills spatial cues from SAM to ground attention in relevant image regions, while preserving language capabilities through a four-stage training pipeline. The approach leverages a ConvNeXt vision backbone and a small LLaMA-1B backbone, with four training stages including SAM-based spatial grounding and instruction fine-tuning. Empirical results on vision-centric benchmarks show AGE-VLM achieves strong grounding and competitive performance against prior efficient VLMs, indicating practical potential for reliable multimodal understanding with limited compute.

Abstract

Large Vision-Language Models (VLMs) rely on effective multimodal alignment between pre-trained vision encoders and Large Language Models (LLMs) to integrate visual and textual information. This paper presents a comprehensive analysis of attention patterns in efficient VLMs, revealing that concatenation-based architectures frequently fail to distinguish between semantically matching and non-matching image-text pairs. This is a key factor for object hallucination in these models. To address this, we introduce Attention-Guided Efficient Vision-Language Models (AGE-VLM), a novel framework that enhances visual grounding through interleaved cross-attention layers to instill vision capabilities in pretrained small language models. This enforces in VLM the ability "look" at the correct image regions by leveraging spatial knowledge distilled from the Segment Anything Model (SAM), significantly reducing hallucination. We validate our approach across different vision-centric benchmarks where our method is better or comparable to prior work on efficient VLMs. Our findings provide valuable insights for future research aimed at achieving enhanced visual and linguistic understanding in VLMs.

Attention Guided Alignment in Efficient Vision-Language Models

TL;DR

The paper addresses object hallucination in efficient vision–language models by diagnosing weak visual grounding in concatenation-based architectures. It introduces AGE-VLM, which interleaves cross-attention in a decoder-only LLM and distills spatial cues from SAM to ground attention in relevant image regions, while preserving language capabilities through a four-stage training pipeline. The approach leverages a ConvNeXt vision backbone and a small LLaMA-1B backbone, with four training stages including SAM-based spatial grounding and instruction fine-tuning. Empirical results on vision-centric benchmarks show AGE-VLM achieves strong grounding and competitive performance against prior efficient VLMs, indicating practical potential for reliable multimodal understanding with limited compute.

Abstract

Large Vision-Language Models (VLMs) rely on effective multimodal alignment between pre-trained vision encoders and Large Language Models (LLMs) to integrate visual and textual information. This paper presents a comprehensive analysis of attention patterns in efficient VLMs, revealing that concatenation-based architectures frequently fail to distinguish between semantically matching and non-matching image-text pairs. This is a key factor for object hallucination in these models. To address this, we introduce Attention-Guided Efficient Vision-Language Models (AGE-VLM), a novel framework that enhances visual grounding through interleaved cross-attention layers to instill vision capabilities in pretrained small language models. This enforces in VLM the ability "look" at the correct image regions by leveraging spatial knowledge distilled from the Segment Anything Model (SAM), significantly reducing hallucination. We validate our approach across different vision-centric benchmarks where our method is better or comparable to prior work on efficient VLMs. Our findings provide valuable insights for future research aimed at achieving enhanced visual and linguistic understanding in VLMs.

Paper Structure

This paper contains 33 sections, 3 equations, 3 figures, 6 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Vision in Vision-Language Models
  4. Attention analysis in VLMs
  5. Attention guided Efficient VLM Approach
  6. Efficient Vision-Language Architecture
  7. Efficient vision backbone.
  8. Efficient LLaMA-1B backbone.
  9. Interleaved cross-attention layers.
  10. Spatial Grounding Distillation using SAM
  11. Training Stages
  12. Stage1: Initial vision-language alignment.
  13. Stage 2: Vision encoder adaptation.
  14. Stage 3: Spatial grounding via knowledge distillation and alignment.
  15. Stage 4: Visually-grounded instruction fine-tuning.
  16. ...and 18 more sections

Figures (3)

  • Figure 1: Object hallucination and mitgation in efficient vision language models. We show that prior work on efficient VLMs fails to localize (top) and correctly count the number of objects (cat) in images. Our attention guided efficient VLMs approach equipped with the knowledge distillation from Segment Anything Model in the cross-attention weights can effectively generate correct responses with explanation grounded in the visual domain.
  • Figure 2: Similarity analysis. Cosine similarly between the hidden states of the images and text tokens of the last self-attention layer in existing efficient architectures. The similarities overlap for non-matching pairs indicating a gap in the alignment of visual signal with text.
  • Figure 3: Overall architecture of our attention-guided efficient vision language model. During training, given the input image and the associated instruction, we perform knowledge distillation from SAM by explicitly aligning the language-conditioned masks with the cross-attention weights of our modified LLaMA-1B backbone.