Zoomer: Adaptive Image Focus Optimization for Black-box MLLM

TL;DR

The paper tackles hallucination and detail loss in real-world visual reasoning by black-box MLLMs under strict token budgets. It introduces Zoomer, a visual prompting framework with three integrated components: a prompt-aware visual emphasizer for region prioritization, a spatial-preserving orchestration schema to maintain layout, and a budget-aware prompting strategy to allocate tokens efficiently. Empirical results across nine benchmarks and multiple MLLMs show up to $27\%$ accuracy gains and up to $67\%$ token savings, demonstrating robust, model-agnostic improvements in token-constrained multimodal understanding. The work provides a principled path toward reliable high-resolution prompting in inaccessible-model settings and outlines directions for future detector-free or edge-aware extensions.

Abstract

Multimodal large language models (MLLMs) such as GPT-4o, Gemini Pro, and Claude 3.5 have enabled unified reasoning over text and visual inputs, yet they often hallucinate in real world scenarios especially when small objects or fine spatial context are involved. We pinpoint two core causes of this failure: the absence of region-adaptive attention and inflexible token budgets that force uniform downsampling, leading to critical information loss. To overcome these limitations, we introduce Zoomer, a visual prompting framework that delivers token-efficient, detail-preserving image representations for black-box MLLMs. Zoomer integrates (1) a prompt-aware emphasis module to highlight semantically relevant regions, (2) a spatial-preserving orchestration schema to maintain object relationships, and (3) a budget-aware strategy to adaptively allocate tokens between global context and local details. Extensive experiments on nine benchmarks and three commercial MLLMs demonstrate that Zoomer boosts accuracy by up to 27% while cutting image token usage by up to 67%. Our approach establishes a principled methodology for robust, resource-aware multimodal understanding in settings where model internals are inaccessible.

Figures (6)

• Figure 1: Illustration of a black-box MLLM's approach to counting cacti in an image. The model identifies two small cacti on the left side and overlooks the single cactus on the right side of the image, arriving at a total of three cacti. The processed prompt highlights specific regions of interest to facilitate the correct object count.
• Figure 2: Illustration of information loss during image processing in black-box MLLMs. The original high-resolution image (4240x2832) is downscaled to meet token limits (1536x1026), leading to the loss of critical details. Cropping to focus on a region of interest (146x246) allows the model to correctly identify the book title as "Webster's II".
• Figure 3: The $\sf{Zoomer}$ framework. Left: Raw Input image (①) and text prompt are processed by $\sf{Zoomer}$ and then fed into a black-box LLM (e.g., GPT-4o) for analysis, resulting in more accurate and detailed responses compared to standard input methods with even token saving. Right: $\sf{Zoomer}$ processes the text to extract key terms and uses a multi-scale emphasizer(§\ref{['subsec:emphasizer']}) with an off-the-shelf object detection model to identify regions of interest (ROIs). The identified ROIs (②) are then processed through a spatial preserving orchestration schema (§\ref{['subsec:orchetration']}) for a filtered emphasized patch (④) and a budget-aware image prompt construction module (§\ref{['subsec:prompting']}) to create a token-efficient prompt within the specified budget. A scaled global view (③) is also generated for potential prompting.
• Figure 4: The example of applying $\sf{Zoomer}$
• Figure 5: The example of different settings of $\sf{Zoomer}$