Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

CARES: Context-Aware Resolution Selector for VLMs

Moshe Kimhi, Nimrod Shabtay, Raja Giryes, Chaim Baskin, Eli Schwartz

TL;DR

CARES introduces a lightweight, model-agnostic preprocessor that predicts the minimal sufficient input resolution for a given image–query pair, reducing token counts and compute prior to vision–language model tokenization. It uses a compact proxy VLM to extract joint image–text features from a low-resolution pass, then classifies among a discrete resolution menu and interpolates a continuous resolution at inference. Ground-truth resolutions are obtained via multi-resolution rollouts and an ANLS-based sufficiency rule, enabling efficient supervision without changing target VLMs. Across five multimodal benchmarks and multiple backbones, CARES achieves 70–85% prefill FLOPs savings with minimal to no drop in accuracy, demonstrating a practical, plug-and-play path to scalable VLM deployment. This front-end resolution control complements post-tokenization compression techniques and opens avenues for finer-grained, query-aware input allocation in multimodal systems.

Abstract

Large vision-language models (VLMs) commonly process images at native or high resolution to remain effective across tasks. This inflates visual tokens ofter to 97-99% of total tokens, resulting in high compute and latency, even when low-resolution images would suffice. We introduce \emph{CARES}-a \textbf{C}ontext-\textbf{A}ware \textbf{R}esolution \textbf{S}elector, a lightweight preprocessing module that, given an image-query pair, predicts the \emph{minimal} sufficient input resolution. CARES uses a compact VLM (350M) to extract features and predict when a target pretrained VLM's response converges to its peak ability to answer correctly. Though trained as a discrete classifier over a set of optional resolutions, CARES interpolates continuous resolutions at inference for fine-grained control. Across five multimodal benchmarks spanning documents and natural images, as well as diverse target VLMs, CARES preserves task performance while reducing compute by up to 80%.

CARES: Context-Aware Resolution Selector for VLMs

TL;DR

CARES introduces a lightweight, model-agnostic preprocessor that predicts the minimal sufficient input resolution for a given image–query pair, reducing token counts and compute prior to vision–language model tokenization. It uses a compact proxy VLM to extract joint image–text features from a low-resolution pass, then classifies among a discrete resolution menu and interpolates a continuous resolution at inference. Ground-truth resolutions are obtained via multi-resolution rollouts and an ANLS-based sufficiency rule, enabling efficient supervision without changing target VLMs. Across five multimodal benchmarks and multiple backbones, CARES achieves 70–85% prefill FLOPs savings with minimal to no drop in accuracy, demonstrating a practical, plug-and-play path to scalable VLM deployment. This front-end resolution control complements post-tokenization compression techniques and opens avenues for finer-grained, query-aware input allocation in multimodal systems.

Abstract

Large vision-language models (VLMs) commonly process images at native or high resolution to remain effective across tasks. This inflates visual tokens ofter to 97-99% of total tokens, resulting in high compute and latency, even when low-resolution images would suffice. We introduce \emph{CARES}-a \textbf{C}ontext-\textbf{A}ware \textbf{R}esolution \textbf{S}elector, a lightweight preprocessing module that, given an image-query pair, predicts the \emph{minimal} sufficient input resolution. CARES uses a compact VLM (350M) to extract features and predict when a target pretrained VLM's response converges to its peak ability to answer correctly. Though trained as a discrete classifier over a set of optional resolutions, CARES interpolates continuous resolutions at inference for fine-grained control. Across five multimodal benchmarks spanning documents and natural images, as well as diverse target VLMs, CARES preserves task performance while reducing compute by up to 80%.
Paper Structure (27 sections, 5 equations, 3 figures, 6 tables, 2 algorithms)

This paper contains 27 sections, 5 equations, 3 figures, 6 tables, 2 algorithms.

Table of Contents

  1. Introduction
  2. Our contributions are as follows:
  3. Related Work
  4. Visual-token sparsification at inference
  5. Training for flexible token budgets
  6. Any-resolution inputs and tiling
  7. Dynamic computation
  8. Adaptive input resolution selection
  9. Extreme compression and design insights
  10. CARES
  11. Problem Definition
  12. Labeling Strategy for Training CARES
  13. Model
  14. From Discrete Supervision to a Continuous Resolution
  15. Continuous inference algorithm.
  16. ...and 12 more sections

Figures (3)

  • Figure 1: Overview of CARES. Given an image and its query, CARES predicts the minimal sufficient input resolution. The image is resized accordingly and, together with the query, passed to a downstream VLM. Coarse queries are routed to lower resolution; fine-grained queries that require more detail trigger higher resolution, which yields more visual tokens in the VLM.
  • Figure 2: Accuracy vs. TTFT for DocVQA with Qwen2.5-VL-72B across native and fixed-resolution settings versus CARES. Bubble size indicates the number of pixels processed by the model.
  • Figure 3: Histogram of the predicted resolutions $\tilde{r}$ by CARES for OCRBench.