Visually Interpretable Subtask Reasoning for Visual Question Answering

TL;DR

This work addresses the interpretability gap in vision-language large language models for visual question answering by introducing VISTAR, a subtask-aware reasoning framework. VISTAR generates Subtask-of-Thought (SoT) rationales via LLM prompts and then fine-tunes MLLMs to produce both textual explanations and object-level visual grounding, eliminating the need for external program execution. The authors create GQA-SoT, an augmented dataset with over 200k SoTs for training and validation, and demonstrate state-of-the-art compositional VQA performance on GQA while improving interpretability metrics such as bounding-box grounding and rationale quality. They also show zero-shot generalization to CRIC and perform comprehensive ablations to validate the contribution of bounding boxes and intermediate steps, highlighting the practical impact of structured, interpretable reasoning in multimodal tasks.

Abstract

Answering complex visual questions like `Which red furniture can be used for sitting?' requires multi-step reasoning, including object recognition, attribute filtering, and relational understanding. Recent work improves interpretability in multimodal large language models (MLLMs) by decomposing tasks into sub-task programs, but these methods are computationally expensive and less accurate due to poor adaptation to target data. To address this, we introduce VISTAR (Visually Interpretable Subtask-Aware Reasoning Model), a subtask-driven training framework that enhances both interpretability and reasoning by generating textual and visual explanations within MLLMs. Instead of relying on external models, VISTAR fine-tunes MLLMs to produce structured Subtask-of-Thought rationales (step-by-step reasoning sequences). Experiments on two benchmarks show that VISTAR consistently improves reasoning accuracy while maintaining interpretability. Our code and dataset will be available at https://github.com/ChengJade/VISTAR.

Figures (8)

• Figure 1: Comparison of standard MLLMs, programmatic reasoning and VISTAR across the output.(a) Existing methods either provide no explanations, lack visual grounding, or have high computational costs. (b) Performance degradation of standard MLLMs (LLaVA-1.5-7B Liu2023ImprovedBW and NVILA-8B Liu2024NVILAEF) when forced to generate explanations alongside answers. 'Exp-GPT-4' evaluates semantic similarity using GPT-4-turboAchiam2023GPT4TR. (c) VISTAR effectively addresses these issues by decomposing the question into structured sub-tasks, providing both visual (bounding box, following the format ($x_l, y_l, x_r, y_r$) where $x_l, y_l$ are the coordinates of the top-left corner and $x_r, y_r$ are the coordinates of the bottom-right corner) and textual rationales without compromising accuracy.
• Figure 2: Overview of VISATR. VISTAR uses an LLM to generate faithful SoTs via in-context learning. Given the input including the query, ground truth, scene graph in an image and sub-task operation sequence inside the dataset (top), LLaMA-3.1-70B-Instruct would output a SoT to answer the query. The generated SoT is then used to fine-tune an MLLM, enabling it to produce both visual (object-level bounding boxes) and textual explanations during inference (bottom).
• Figure 3: Quantitative results of visual explanations. (a) Comparison of answer correctness and object localization accuracy (measured by IoU when the answer is correct). (b-d) Precision & recall at different IoU thresholds (0.5, 0.75, 0.95). VISTAR consistently outperforms NVILA-8B in both accuracy and object-level visual grounding, demonstrating improved interpretability and localization quality.
• Figure 4: Examples of successful and failed cases on human evaluation. We present some successful and failed cases for our SoT prediction based on VISTAR.
• Figure 5: Consistency evaluation between answers and reasoning steps. (a) Ablation results without intermediate answer supervision, evaluated by GPT-4-turbo over operation and arguments with answers (b) Results from the full training procedure, evaluated by GPT-4-turbo over operation and arguments with answers (c) Human evaluation results, assessing the consistency of predicted sub-tasks with their arguments and intermediate results relative to the final answers. Notably, 'T' in the figure means true, 'F' means false and 'TT' indicates both the sub-task and intermediate answer are correct while 'TF', 'FT', and 'FF' represent cases where either the sub-task, intermediate answer, or both are incorrect.