AdaptVision: Efficient Vision-Language Models via Adaptive Visual Acquisition

TL;DR

AdaptVision tackles the high computational cost of vision-language models by enabling adaptive, coarse-to-fine visual token acquisition through a bounding-box tool. It casts the problem as reinforcement learning and introduces Decoupled Turn Policy Optimization to separately optimize tool usage and answer accuracy, improving credit assignment and training stability. Across diverse VQA benchmarks, AdaptVision achieves higher accuracy with substantially fewer visual tokens and modest latency, outperforming fixed-ratio and other dynamic methods. The work advances efficient, active visual reasoning for VLMs and suggests a path toward per-sample token economy in visual question answering.

Abstract

Vision-Language Models (VLMs) have achieved remarkable success in visual question answering tasks, but their reliance on large numbers of visual tokens introduces significant computational overhead. While existing efficient VLM approaches reduce visual tokens through fixed-ratio compression, they operate passively and lack the ability to adapt to varying task requirements. This motivates a fundamental question: Can VLMs autonomously determine the minimum number of visual tokens required for each sample? Inspired by human active vision mechanisms, we introduce AdaptVision, an efficient VLM paradigm that enables adaptive visual token acquisition through a coarse-to-fine approach. Our model initially processes compressed visual tokens from low-resolution images and selectively acquires additional visual information by invoking a bounding box tool to crop key regions when necessary. We train AdaptVision using a reinforcement learning framework that carefully balances accuracy and efficiency. Central to our approach is Decoupled Turn Policy Optimization (DTPO), which decouples the learning objective into two components: (1) tool learning, which optimizes correct tool utilization, and (2) accuracy improvement, which refines the generated responses to improve answer correctness. Based on this formulation, we further decouple advantage estimation by computing separate advantages for tokens associated with each objective. This formulation enables more effective optimization for AdaptVision compared to vanilla GRPO. Comprehensive experiments across multiple VQA benchmarks demonstrate that AdaptVision achieves superior performance while consuming substantially fewer visual tokens than state-of-the-art efficient VLM methods.

Figures (9)

• Figure 1: Our key motivations and AdaptVision performance and efficiency.Top: Coarse-to-fine. Human visual attention mechanisms first guide the search for question-relevant regions in images, which are then subjected to detailed analysis. Down: AdaptVision achieves superior performance with significantly fewer visual tokens than previous efficient VLM methods.
• Figure 2: FrameWork of AdaptVision. AdaptVision first processes a 1/4-resolution image. The model then decides whether to answer directly or invoke the bounding box tool to crop a high-resolution region for further analysis before generating the final answer.
• Figure 3: Demonstration of vanilla GRPO and our DTPO. Our DTPO (1) decomposes the policy loss by turns to separately optimize tool and answer tokens, and (2) computes distinct advantages for tool and outcome rewards, enabling balanced optimization and precise credit assignment.
• Figure 4: Comparison of Inference Time. (1) Compared to the vanilla model and VisionThink†, AdaptVision demonstrates significantly reduced inference time due to reduced visual token usage. (2) While AdaptVision requires additional generated tokens for reasoning and tool calls compared to the down-sample model, the resulting increase in inference time remains acceptable.
• Figure 5: Policy-training comparison: (a) The influence of reward design. (b) GRPO vs. DTPO.