ERGO: Efficient High-Resolution Visual Understanding for Vision-Language Models

Abstract

Efficient processing of high-resolution images is crucial for real-world vision-language applications. However, existing Large Vision-Language Models (LVLMs) incur substantial computational overhead due to the large number of vision tokens. With the advent of "thinking with images" models, reasoning now extends beyond text to the visual domain. This capability motivates our two-stage "coarse-to-fine" reasoning pipeline: first, a downsampled image is analyzed to identify task-relevant regions; then, only these regions are cropped at full resolution and processed in a subsequent reasoning stage. This approach reduces computational cost while preserving fine-grained visual details where necessary. A major challenge lies in inferring which regions are truly relevant to a given query. Recent related methods often fail in the first stage after input-image downsampling, due to perception-driven reasoning, where clear visual information is required for effective reasoning. To address this issue, we propose ERGO (Efficient Reasoning & Guided Observation) that performs reasoning-driven perception-leveraging multimodal context to determine where to focus. Our model can account for perceptual uncertainty, expanding the cropped region to cover visually ambiguous areas for answering questions. To this end, we develop simple yet effective reward components in a reinforcement learning framework for coarse-to-fine perception. Across multiple datasets, our approach delivers higher accuracy than the original model and competitive methods, with greater efficiency. For instance, ERGO surpasses Qwen2.5-VL-7B on the V* benchmark by 4.7 points while using only 23% of the vision tokens, achieving a 3x inference speedup. The code and models can be found at: https://github.com/nota-github/ERGO.

Figures (12)

• Figure 1: Comparison with prior work on high-resolution visual reasoning. The yellow box marks the target object, which becomes indiscernible after input-image downsampling. (a) DeepEyes zheng2025deepeyes succeeds when the object remains discernible, but at the cost of a large number of vision tokens. (b) DeepEyes zheng2025deepeyes fails when the object is indiscernible at low resolution, where fewer vision tokens are available. (c) Our ERGO performs reasoning-driven perception, correctly answering the question even on low-resolution images.
• Figure 2: Overview of RL-based training pipeline. The blue background highlights the components of the proposed TCE reward. The purple background highlights the conventional rewards adopted by most reasoning LVLMs.
• Figure 3: Analysis of query-relevant GT regions in training data. Most GT regions span less than 60% of the full image area.
• Figure 4: Performance-efficiency trade-off on the V* benchmark. The total number of vision tokens is the sum of the tokens from the downsampled original image and those from the high-resolution cropped image.
• Figure 5: Evaluation of model robustness under target-object masking.