The Impact of Image Resolution on Biomedical Multimodal Large Language Models

TL;DR

The study addresses how image resolution impacts biomedical multimodal LLMs by showing that native-resolution training and inference preserve fine-grained biomedical details and substantially improve task performance. It demonstrates that resolution misalignment between training and inference can cause large performance drops, and that mixed-resolution training offers a practical compromise balancing accuracy and compute. Across seven diverse biomedical datasets, native-resolution methods yield improvements up to approximately 6.8% in accuracy, while misalignment can erase much of these gains; mixed-resolution training reduces average losses to around 1.0% and sustains native-resolution advantages during inference. The work provides actionable guidance for users and developers: prioritize native-resolution inference when possible, and adopt balanced mixed-resolution training to handle dataset variability and compute constraints, ultimately enhancing the deployment of biomedical MLLMs in clinical and research settings.

Abstract

Imaging technologies are fundamental to biomedical research and modern medicine, requiring analysis of high-resolution images across various modalities. While multimodal large language models (MLLMs) show promise for biomedical image analysis, most are designed for low-resolution images from general-purpose datasets, risking critical information loss. We investigate how image resolution affects MLLM performance in biomedical applications and demonstrate that: (1) native-resolution training and inference significantly improve performance across multiple tasks, (2) misalignment between training and inference resolutions severely degrades performance, and (3) mixed-resolution training effectively mitigates misalignment and balances computational constraints with performance requirements. Based on these findings, we recommend prioritizing native-resolution inference and mixed-resolution datasets to optimize biomedical MLLMs for transformative impact in scientific research and clinical applications.

Figures (5)

• Figure 1: Biomedical MLLMs process images across modalities like radiology, pathology, and microscopy, however standard MLLMs are not designed to handle varying image resolution or high resolutions. This paper argues for native-resolution biomedical MLLMs. The image encoder processes arbitrary-resolution images into tokens dynamically aligned with language tokens, preserving high-frequency details critical for biomedical tasks. The textual output presents an example pathology question-answering task.
• Figure 2: Comparison of image patches at native resolution (left) and low resolution. Downsampling often removes high-frequency details, such as fine textural details in protein staining, that are relevant for accurate interpretation.
• Figure 3: Performance comparison of native-resolution and low-resolution training and inference. Native-resolution training and inference achieve superior results, emphasizing the importance of resolution fidelity.
• Figure 4: Native-resolution inference improves off-the-shelf models. Qwen2-VL 7B shows consistent gains on radiological and pathological question-answering tasks with native-resolution input.
• Figure 5: Comparing low-resolution, high-resolution, and images resampled from low-resolution images.