VOILA: Value-of-Information Guided Fidelity Selection for Cost-Aware Multimodal Question Answering

TL;DR

VOILA addresses the high cost of multimodal inputs by introducing a pre-retrieval fidelity selection framework that predicts the minimum necessary visual fidelity from a user query. It combines a two-stage predictor (question-conditioned gradient-boosted success scoring followed by isotonic calibration) with a Value-of-Information-based decision rule to choose the lowest-cost fidelity that yields a worthwhile accuracy gain, before any image retrieval or VLM execution. Evaluations across five datasets and six Vision-Language Models show 50–60% reductions in retrieval cost while maintaining 90–95% of full-resolution accuracy, with strong out-of-distribution generalization and minimal overhead. The work also provides a formal regret bound linked to calibration quality, demonstrating that adaptive pre-retrieval information acquisition is a principled and practical complement to post-retrieval model routing for resource-constrained multimodal AI systems.

Abstract

Despite significant costs from retrieving and processing high-fidelity visual inputs, most multimodal vision-language systems operate at fixed fidelity levels. We introduce VOILA, a framework for Value-Of-Information-driven adaptive fidelity selection in Visual Question Answering (VQA) that optimizes what information to retrieve before model execution. Given a query, VOILA uses a two-stage pipeline: a gradient-boosted regressor estimates correctness likelihood at each fidelity from question features alone, then an isotonic calibrator refines these probabilities for reliable decision-making. The system selects the minimum-cost fidelity maximizing expected utility given predicted accuracy and retrieval costs. We evaluate VOILA across three deployment scenarios using five datasets (VQA-v2, GQA, TextVQA, LoCoMo, FloodNet) and six Vision-Language Models (VLMs) with 7B-235B parameters. VOILA consistently achieves 50-60% cost reductions while retaining 90-95% of full-resolution accuracy across diverse query types and model architectures, demonstrating that pre-retrieval fidelity selection is vital to optimize multimodal inference under resource constraints.

Figures (5)

• Figure 1: Given a user query, VOILA predicts the minimum visual fidelity required before retrieval
• Figure 2: Empirical evidence with Pixtral-12B on VQA-v2 motivating VOILA's design
• Figure 3: Calibrated vs. uncalibrated probabilities
• Figure 4: Accuracy--cost Pareto frontiers across datasets and models. Each subplot plots accuracy versus normalized retrieval cost (upper left is better) for a dataset--model pair, comparing VOILA to fixed-fidelity baselines. Dashed curves denote Pareto frontiers. Across datasets and model scales, VOILA lies on or near the frontier, achieving near--full-resolution accuracy at 50--60% lower average cost, while no single fixed fidelity is Pareto-optimal.
• Figure 5: Learned fidelity selection distributions under VOI routing. Each subplot reports the percentage of queries routed to each input fidelity by VOILA, across four datasets (rows) and six vision--language models (columns). Low-cost representations (captions and low-resolution thumbnails) dominate across all settings, while higher-cost fidelities (JPEG Q1/Q10 and full-resolution images) are selected selectively when expected utility justifies their acquisition cost.

Theorems & Definitions (9)

• Proposition 3.1: Exact Optimality
• proof
• Definition 3.2: Uniform Calibration Error
• Theorem 3.3: Utility Regret Bound
• proof
• Lemma 3.4: Empirical $\!\to\!$ Population Calibration
• proof : Proof Sketch
• Corollary 3.5: Regret of Greedy Sequential Policy
• proof