Question Aware Vision Transformer for Multimodal Reasoning

TL;DR

This paper tackles the misalignment between vision encodings and user queries in vision-language models by introducing QA-ViT, a lightweight, model-agnostic approach that conditions the vision transformer on the question. It computes a question representation, projects it per ViT layer, and fuses it into the frozen vision backbone via a late-fusion, cross-attention mechanism, preserving visual understanding while guiding attention to query-relevant regions. Extensive experiments across ViT+T5 and state-of-the-art VL models (BLIP2, InstructBLIP, LLaVA-1.5) on general and scene-text tasks show consistent improvements, including zero-shot settings, with ablations highlighting late fusion, question-encoder choice, and multi-task training as key factors. The results indicate QA-ViT’s strong potential to enhance OCR-aware visual reasoning in a broad range of VL applications, while remaining compatible with diverse architectures and scales.

Abstract

Vision-Language (VL) models have gained significant research focus, enabling remarkable advances in multimodal reasoning. These architectures typically comprise a vision encoder, a Large Language Model (LLM), and a projection module that aligns visual features with the LLM's representation space. Despite their success, a critical limitation persists: the vision encoding process remains decoupled from user queries, often in the form of image-related questions. Consequently, the resulting visual features may not be optimally attuned to the query-specific elements of the image. To address this, we introduce QA-ViT, a Question Aware Vision Transformer approach for multimodal reasoning, which embeds question awareness directly within the vision encoder. This integration results in dynamic visual features focusing on relevant image aspects to the posed question. QA-ViT is model-agnostic and can be incorporated efficiently into any VL architecture. Extensive experiments demonstrate the effectiveness of applying our method to various multimodal architectures, leading to consistent improvement across diverse tasks and showcasing its potential for enhancing visual and scene-text understanding.

Figures (7)

• Figure 1: Question-Aware Vision Encoding. Comparative illustrations for VQAv2 (upper) and TextVQA (lower) predictions of ViT+T5 and QA-ViT+T5 VL models. Employing GradCAM highlights the focus areas with respect to key terms in the posed questions. This vividly demonstrates the motivation behind QA-ViT: enhancing ViT with the question enables it to focus on the relevant image aspects, resulting in more accurate predictions.
• Figure 2: Method overview. A high-level illustration of the QA-ViT (highlighted in orange) incorporated into a general VL architecture (depicted in blue). This is achieved by encoding the question $Q$ into features $F_Q$, which are fused into the vision encoder, resulting in question-aware visual features $F_{VQ}$.
• Figure 3: Textual representations fusing. Left: General scheme of the ViT encoder. Right: Zoom in to our fusing mechanism in one of the top-L self-attention layers. The $M$ visual features from the previous layer $F_{V}$, are concatenated with $K$ textual features $F_Q$ and fed into the frozen self-attention mechanism to obtain $M$ text-attended visual representations $F_{VQ}'$. Next, a parallel gated projection obtains the question-aware visual features of $F_{VQ}$.
• Figure 4: Paying attention to details in visual question answering. Representative examples require answering questions regarding subtle or less conspicuous image details (zoomed-in) from VQAv2 and TextVQA datasets. Each sample includes an image-question pair alongside predictions from ViT+T5 and QA-ViT +T5, where green indicates correct predictions and red indicates incorrect ones.
• Figure 5: QA-ViT effectiveness analysis. Comparison of the trends in error rate reduction of QA-ViT in VQA$^\text{T}$ and VQA$^\text{v2}$ as the language model is scaled up. The relative performance improvements of our approach are more consistent across model scales in the former. These trends are attributed to each dataset's different question types' composition, where VQA$^\text{T}$ exhibits more questions focusing on non-salient and overlooked elements.