Task-Aware Resolution Optimization for Visual Large Language Models

TL;DR

This work tackles the fixed-resolution limitation in vision-language models by proposing a task-aware resolution optimization framework. It introduces two interpretable heuristics—image complexity and uncertainty variance—and a simple empirical formula, Reso(T) = $Reso_0 \cdot (1 + k \cdot C(T) \cdot V(T))$, to select a task-specific optimal input resolution without exhaustive retraining. For adaptation, the paper presents a parameter-efficient post-training (PEFT) approach that extends the visual input resolution by updating only the visual-position embeddings, projector, and LoRA adapters, while keeping the rest of the model frozen. Empirical evaluations on eight vision-language tasks using LLaVA-1.5-7B show that task-wise resolution selection combined with PEFT outperforms fixed-resolution baselines and many state-of-the-art approaches, with ablations confirming the necessity of each PEFT component. Case studies and analyses highlight the practical impact of image complexity and question difficulty on resolution choice, providing a scalable pathway for real-world VLLM deployment.

Abstract

Real-world vision-language applications demand varying levels of perceptual granularity. However, most existing visual large language models (VLLMs), such as LLaVA, pre-assume a fixed resolution for downstream tasks, which leads to subpar performance. To address this problem, we first conduct a comprehensive and pioneering investigation into the resolution preferences of different vision-language tasks, revealing a correlation between resolution preferences with image complexity, and uncertainty variance of the VLLM at different image input resolutions. Building on this insight, we propose an empirical formula to determine the optimal resolution for a given vision-language task, combining these two factors. Second, based on rigorous experiments, we propose a novel parameter-efficient fine-tuning technique to extend the visual input resolution of pre-trained VLLMs to the identified optimal resolution. Extensive experiments on various vision-language tasks validate the effectiveness of our method.

Figures (7)

• Figure 1: Resolution preference across eight tasks; ★ marks the optimal resolution for each task.
• Figure 2: Our method comprises two components: the first component identifies the optimal image input resolution for a given vision-language task (depicted in green), while the second component adapts the VLLM to the selected image input resolution (depicted in blue).
• Figure 3: Correlation of heuristic metrics with preferred task resolution. The product of $C(T)$ and $V(T)$ exhibits a more consistent correlation compared to individual heuristics. All metrics are normalized for visualization.
• Figure 4: We select three reference tasks with images in different levels of complexity to optimize the hyperparameter in Equation \ref{['eq:empirical-formula']}.
• Figure 5: Applying the empirical formula to determine the optimal resolution for vision-language tasks.