Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Weaving Context Across Images: Improving Vision-Language Models through Focus-Centric Visual Chains

Juntian Zhang, Chuanqi cheng, Yuhan Liu, Wei Liu, Jian Luan, Rui Yan

TL;DR

The paper confronts the challenge that vision–language models struggle with multi-image inputs due to cross-image correlations and information fragmentation. It introduces the Focus-Centric Visual Chain for progressive, sub-question–driven reasoning and the Focus-Centric Data Synthesis framework to generate large-scale, high-quality reasoning data, resulting in the VISC-150K dataset. Empirical results across seven benchmarks show consistent improvements across open-source baselines and state-of-the-art gains on several tasks, including substantial gains when combined with strong models. The work provides a scalable, data-centric pathway to robust multi-image reasoning and lays groundwork for transferability to diverse visual domains and tasks.

Abstract

Vision-language models (VLMs) achieve remarkable success in single-image tasks. However, real-world scenarios often involve intricate multi-image inputs, leading to a notable performance decline as models struggle to disentangle critical information scattered across complex visual features. In this work, we propose Focus-Centric Visual Chain, a novel paradigm that enhances VLMs'perception, comprehension, and reasoning abilities in multi-image scenarios. To facilitate this paradigm, we propose Focus-Centric Data Synthesis, a scalable bottom-up approach for synthesizing high-quality data with elaborate reasoning paths. Through this approach, We construct VISC-150K, a large-scale dataset with reasoning data in the form of Focus-Centric Visual Chain, specifically designed for multi-image tasks. Experimental results on seven multi-image benchmarks demonstrate that our method achieves average performance gains of 3.16% and 2.24% across two distinct model architectures, without compromising the general vision-language capabilities. our study represents a significant step toward more robust and capable vision-language systems that can handle complex visual scenarios.

Weaving Context Across Images: Improving Vision-Language Models through Focus-Centric Visual Chains

TL;DR

The paper confronts the challenge that vision–language models struggle with multi-image inputs due to cross-image correlations and information fragmentation. It introduces the Focus-Centric Visual Chain for progressive, sub-question–driven reasoning and the Focus-Centric Data Synthesis framework to generate large-scale, high-quality reasoning data, resulting in the VISC-150K dataset. Empirical results across seven benchmarks show consistent improvements across open-source baselines and state-of-the-art gains on several tasks, including substantial gains when combined with strong models. The work provides a scalable, data-centric pathway to robust multi-image reasoning and lays groundwork for transferability to diverse visual domains and tasks.

Abstract

Vision-language models (VLMs) achieve remarkable success in single-image tasks. However, real-world scenarios often involve intricate multi-image inputs, leading to a notable performance decline as models struggle to disentangle critical information scattered across complex visual features. In this work, we propose Focus-Centric Visual Chain, a novel paradigm that enhances VLMs'perception, comprehension, and reasoning abilities in multi-image scenarios. To facilitate this paradigm, we propose Focus-Centric Data Synthesis, a scalable bottom-up approach for synthesizing high-quality data with elaborate reasoning paths. Through this approach, We construct VISC-150K, a large-scale dataset with reasoning data in the form of Focus-Centric Visual Chain, specifically designed for multi-image tasks. Experimental results on seven multi-image benchmarks demonstrate that our method achieves average performance gains of 3.16% and 2.24% across two distinct model architectures, without compromising the general vision-language capabilities. our study represents a significant step toward more robust and capable vision-language systems that can handle complex visual scenarios.
Paper Structure (25 sections, 9 equations, 8 figures, 3 tables)

This paper contains 25 sections, 9 equations, 8 figures, 3 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Vision-Language Models
  4. Reasoning of LLMs and VLMs
  5. Methodology
  6. Focus-Centric Visual Chain
  7. Focus-Centric Data Synthesis
  8. Feature Extraction.
  9. Pair Connection.
  10. Relevance Annotation.
  11. Question Generation.
  12. Experiments
  13. Experimental Setup
  14. Baselines
  15. Benchmarks
  16. ...and 10 more sections

Figures (8)

  • Figure 1: A multi-image QA example: Using Focus-Centric Visual Chain, our model LLaVA-OneVision-VISC successfully answers a question that both GPT-4o and LLaVA-OneVision fail to solve correctly.
  • Figure 2: The Focus-Centric Data Synthesis framework consists of four stages: Feature Extraction processes raw visual input, extracting object-level attributes and interactions into structured image profiles. Pair Connection links related image nodes based on their profiles. Relevance Annotation identifies and annotates relationships between nodes from temporal, spatial, and semantic perspectives. Question Generation utilizes the extracted image profiles and relationship annotations to construct multi-image reasoning paths and corresponding questions.
  • Figure 3: The impact of dataset scale on LLaVA-OneVision's performance across MMIU and MuirBench benchmarks. As the data scale increases, the model's accuracy progressively improves.
  • Figure 4: The accuracy comparison of LLaVA-OneVision on 12 MuirBench sub-tasks with and without being fine-tuned on VISC-150K.
  • Figure 5: The distribution of task accuracy for LLaVA-OneVision based models across varying numbers of input images, grouped into eight buckets from 1 to 16 images with an interval of 2.
  • ...and 3 more figures