Towards Perceiving Small Visual Details in Zero-shot Visual Question Answering with Multimodal LLMs

TL;DR

The paper tackles the problem that zero-shot visual question answering with multimodal LLMs is sensitive to the size of the visual detail being queried. It quantitatively demonstrates a size-based perception gap and proves causality via human cropping, then introduces five automatic visual cropping methods (three external localization-based and two native, model-driven) to mitigate the gap, primarily using BLIP-2 as a proof-of-concept. Across five VQA benchmarks, cropping improves zero-shot accuracy, with notable gains on detail-focused tasks and competitive runtime characteristics, suggesting cropping as a practical enhancement for detail-sensitive VQA scenarios. The work provides actionable techniques and releases code to foster further research into MLLM perceptual limitations and mitigation strategies.

Abstract

Multimodal Large Language Models (MLLMs) have recently achieved promising zero-shot accuracy on visual question answering (VQA) -- a fundamental task affecting various downstream applications and domains. Given the great potential for the broad use of these models, it is important to investigate their limitations in dealing with different image and question properties. In this work, we investigate whether MLLMs can perceive small details as well as large details in images. In particular, we show that their zero-shot accuracy in answering visual questions is very sensitive to the size of the visual subject of the question, declining up to 46% with size. Furthermore, we show that this effect is causal by observing that human visual cropping can significantly mitigate their sensitivity to size. Inspired by the usefulness of human cropping, we then propose five automatic visual cropping methods -- leveraging either external localization models or the decision process of the given MLLM itself -- as inference time mechanisms to improve the zero-shot performance of MLLMs. We study their effectiveness on four popular VQA datasets, and a subset of the VQAv2 dataset tailored towards fine visual details. Our findings suggest that MLLMs should be used with caution in detail-sensitive VQA applications, and that visual cropping is a promising direction to improve their zero-shot performance. To facilitate further investigation of MLLMs' behaviors, our code and data are publicly released.

Figures (12)

• Figure 1: The effect of visual cropping on the probability of answers predicted by BLIP-2 FlanT5$_\mathrm{XL}$ zero-shot VQA model. The x-axis represents the relative crop size around the relevant visual subject of the question (x-axis labels are indices to the respective cropped images displayed under each plot that the model sees at each step). The model gradually finds the correct answer as visual cropping allows it to look closer and thereby better perceive small visual details.
• Figure 2: Illustration of the proposed visual cropping approach applied to two variants of BLIP2.
• Figure 3: Illustration of att-CROP method, where $\sigma$ denotes $ReLU$, $H, L$ the heads and layers of the Transformer, $T$ the query tokens in Q-former, $s$ the last input token for LLM which is used to compute the $l$.
• Figure 4: Illustration of the post-processing for the native cropping methods. After retaining the top $k$ image patches, we select the component with the largest sum value. The bounding box is the smallest rectangle containing that component (green box).
• Figure 5: Examples of success and failure of native cropping methods in correcting the mistakes of BLIP-2 FlanT5$_\mathrm{XL}$ on TextVQA. (Left, middle) BLIP-2 is able to recognize the salient area although fail to answer the question, and therefore native cropping is effective. (Right) BLIP-2 entirely fails to recognize the salient area and therefore native cropping is ineffective.