Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Global Semantic-Guided Sub-image Feature Weight Allocation in High-Resolution Large Vision-Language Models

Yuxuan Liang, Xu Li, Xiaolei Chen, Haotian Chen, Yi Zheng, Chenghang Lai, Bin Li, Xiangyang Xue

TL;DR

This work tackles high-resolution image understanding in LVLMs by addressing the limitations of uniform sub-image processing. It introduces the Global Semantic-guided Weight Allocator (GSWA), which uses inter-sub-image self-attention over <cls> tokens to estimate information density and weight sub-images accordingly, forming SleighVL when integrated into InternVL2-2B. Experiments across general, real-world, text-rich VQA, and other benchmarks show SleighVL outperforms smaller-param models and remains competitive with larger models, validating the effectiveness of density-aware sub-image weighting. The approach offers a scalable, context-aware pathway for efficient high-resolution multimodal processing.

Abstract

As the demand for high-resolution image processing in Large Vision-Language Models (LVLMs) grows, sub-image partitioning has become a popular approach for mitigating visual information loss associated with fixed-resolution processing. However, existing partitioning methods uniformly process sub-images, resulting in suboptimal image understanding. In this work, we reveal that the sub-images with higher semantic relevance to the entire image encapsulate richer visual information for preserving the model's visual understanding ability. Therefore, we propose the Global Semantic-guided Weight Allocator (GSWA) module, which dynamically allocates weights to sub-images based on their relative information density, emulating human visual attention mechanisms. This approach enables the model to focus on more informative regions, overcoming the limitations of uniform treatment. We integrate GSWA into the InternVL2-2B framework to create SleighVL, a lightweight yet high-performing model. Extensive experiments demonstrate that SleighVL outperforms models with comparable parameters and remains competitive with larger models. Our work provides a promising direction for more efficient and contextually aware high-resolution image processing in LVLMs, advancing multimodal system development.

Global Semantic-Guided Sub-image Feature Weight Allocation in High-Resolution Large Vision-Language Models

TL;DR

This work tackles high-resolution image understanding in LVLMs by addressing the limitations of uniform sub-image processing. It introduces the Global Semantic-guided Weight Allocator (GSWA), which uses inter-sub-image self-attention over <cls> tokens to estimate information density and weight sub-images accordingly, forming SleighVL when integrated into InternVL2-2B. Experiments across general, real-world, text-rich VQA, and other benchmarks show SleighVL outperforms smaller-param models and remains competitive with larger models, validating the effectiveness of density-aware sub-image weighting. The approach offers a scalable, context-aware pathway for efficient high-resolution multimodal processing.

Abstract

As the demand for high-resolution image processing in Large Vision-Language Models (LVLMs) grows, sub-image partitioning has become a popular approach for mitigating visual information loss associated with fixed-resolution processing. However, existing partitioning methods uniformly process sub-images, resulting in suboptimal image understanding. In this work, we reveal that the sub-images with higher semantic relevance to the entire image encapsulate richer visual information for preserving the model's visual understanding ability. Therefore, we propose the Global Semantic-guided Weight Allocator (GSWA) module, which dynamically allocates weights to sub-images based on their relative information density, emulating human visual attention mechanisms. This approach enables the model to focus on more informative regions, overcoming the limitations of uniform treatment. We integrate GSWA into the InternVL2-2B framework to create SleighVL, a lightweight yet high-performing model. Extensive experiments demonstrate that SleighVL outperforms models with comparable parameters and remains competitive with larger models. Our work provides a promising direction for more efficient and contextually aware high-resolution image processing in LVLMs, advancing multimodal system development.
Paper Structure (17 sections, 8 equations, 4 figures, 6 tables)

This paper contains 17 sections, 8 equations, 4 figures, 6 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Large Vision-Language Models
  4. High-Resolution Processing in LVLMs
  5. Preliminary Observations
  6. Methods
  7. Overview
  8. Global Semantic-guided Weight Allocator
  9. Experiments
  10. Implementation Details
  11. General Benchmark Results
  12. Realworld Benchmark Results
  13. Text-rich VQA Benchmark Results
  14. Others Benchmark Results
  15. Ablation Study
  16. ...and 2 more sections

Figures (4)

  • Figure 1: An example of human visual saliency analysis highlighting the areas of interest. (a) illustrates an image of a horse-drawn sled carrying four individuals in a snowy landscape, and (b) presents the human visual saliency map generated by SalGAN pan2017salgan, which highlights the regions of the image with high saliency and information density that attract human visual attention.
  • Figure 2: Radar chart comparing our model with existing popular LVLMs of similar parameter scales across ten benchmarks.
  • Figure 3: Two case studies to examine the semantic similarity distribution between each sub-image and the global image. (a) illustrates the degree of similarity between each sub-image and the global image semantics "the player and football" in a football scene, and (b) illustrates the degree of similarity between each sub-image and the global image semantics "the basketball players and court" in a basketball scene.
  • Figure 4: The workflow of the proposed method. (a) describes the overall framework of SleighVL. (b) shows the design details of the global semantic-guided weight allocator.