Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

ViCor: Bridging Visual Understanding and Commonsense Reasoning with Large Language Models

Kaiwen Zhou, Kwonjoon Lee, Teruhisa Misu, Xin Eric Wang

TL;DR

This work probes how pre-trained vision-language models and large language models can jointly tackle visual commonsense reasoning without in-domain fine-tuning. By classifying VCR problems into Visual Commonsense Understanding (VCU) and Visual Commonsense Inference (VCI), ViCor uses LLMs to steer problem analysis and VLMs to provide perceptual input, including active gathering of visual cues when needed. The framework demonstrates strong performance on VCR and A-OKVQA as a training-free approach and offers insights into when to rely on perception versus reasoning. The findings highlight the potential of modular, text-mediated cross-modal collaboration, while acknowledging caption-induced information loss and risks of LLM hallucination, guiding future end-to-end or fine-tuning strategies.

Abstract

In our work, we explore the synergistic capabilities of pre-trained vision-and-language models (VLMs) and large language models (LLMs) on visual commonsense reasoning (VCR) problems. We find that VLMs and LLMs-based decision pipelines are good at different kinds of VCR problems. Pre-trained VLMs exhibit strong performance for problems involving understanding the literal visual content, which we noted as visual commonsense understanding (VCU). For problems where the goal is to infer conclusions beyond image content, which we noted as visual commonsense inference (VCI), VLMs face difficulties, while LLMs, given sufficient visual evidence, can use commonsense to infer the answer well. We empirically validate this by letting LLMs classify VCR problems into these two categories and show the significant difference between VLM and LLM with image caption decision pipelines on two subproblems. Moreover, we identify a challenge with VLMs' passive perception, which may miss crucial context information, leading to incorrect reasoning by LLMs. Based on these, we suggest a collaborative approach, named ViCor, where pre-trained LLMs serve as problem classifiers to analyze the problem category, then either use VLMs to answer the question directly or actively instruct VLMs to concentrate on and gather relevant visual elements to support potential commonsense inferences. We evaluate our framework on two VCR benchmark datasets and outperform all other methods that do not require in-domain fine-tuning.

ViCor: Bridging Visual Understanding and Commonsense Reasoning with Large Language Models

TL;DR

This work probes how pre-trained vision-language models and large language models can jointly tackle visual commonsense reasoning without in-domain fine-tuning. By classifying VCR problems into Visual Commonsense Understanding (VCU) and Visual Commonsense Inference (VCI), ViCor uses LLMs to steer problem analysis and VLMs to provide perceptual input, including active gathering of visual cues when needed. The framework demonstrates strong performance on VCR and A-OKVQA as a training-free approach and offers insights into when to rely on perception versus reasoning. The findings highlight the potential of modular, text-mediated cross-modal collaboration, while acknowledging caption-induced information loss and risks of LLM hallucination, guiding future end-to-end or fine-tuning strategies.

Abstract

In our work, we explore the synergistic capabilities of pre-trained vision-and-language models (VLMs) and large language models (LLMs) on visual commonsense reasoning (VCR) problems. We find that VLMs and LLMs-based decision pipelines are good at different kinds of VCR problems. Pre-trained VLMs exhibit strong performance for problems involving understanding the literal visual content, which we noted as visual commonsense understanding (VCU). For problems where the goal is to infer conclusions beyond image content, which we noted as visual commonsense inference (VCI), VLMs face difficulties, while LLMs, given sufficient visual evidence, can use commonsense to infer the answer well. We empirically validate this by letting LLMs classify VCR problems into these two categories and show the significant difference between VLM and LLM with image caption decision pipelines on two subproblems. Moreover, we identify a challenge with VLMs' passive perception, which may miss crucial context information, leading to incorrect reasoning by LLMs. Based on these, we suggest a collaborative approach, named ViCor, where pre-trained LLMs serve as problem classifiers to analyze the problem category, then either use VLMs to answer the question directly or actively instruct VLMs to concentrate on and gather relevant visual elements to support potential commonsense inferences. We evaluate our framework on two VCR benchmark datasets and outperform all other methods that do not require in-domain fine-tuning.
Paper Structure (27 sections, 10 equations, 4 figures, 5 tables)

This paper contains 27 sections, 10 equations, 4 figures, 5 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Visual Commonsense Reasoning
  4. Large Language Models for Vision-and-Language Tasks
  5. Visual Commonsense Reasoning
  6. Problem Categorization
  7. Visual Commonsense Understanding
  8. Visual Commonsense Inference
  9. Problem Formulation
  10. The ViCor Framework
  11. Large Language Models as VCR Reasoner
  12. Large Language Models as VCR Problem Classifier
  13. Large Language Models as VLM Commander
  14. Experiments
  15. Datasets
  16. ...and 12 more sections

Figures (4)

  • Figure 1: Two examples demonstrating different kinds of visual commonsense reasonings require different model capabilities. Upper: Visual commonsense understanding (VCU) requires the model to understand high-level concepts and attributes such as actions, events, relations, etc, which pre-trained VLMs can achieve via image-text alignment (ITA). Lower: Visual commonsense inference (VCI) requires the model to generate conclusions or explanations based on input image. Overlooking visual clues can result in erroneous conclusions. LLMs steer VLMs in discovering vital visual cues for answer support. The LLM employs the top ITA-scored visual clue (e.g.,"It is cloudy.") to perform commonsense inference.
  • Figure 2: Our ViCor framework. Given a visual commonsense reasoning problem and a caption, our framework will leverage LLM to perform initial reasoning and confidence check. If the reasoning is not confident, the LLM will perform problem classification and acquire visual information according to the problem type. $*$Note that the final reasoning takes the question and the caption as input as well.
  • Figure 3: Three simplified prompt examples demonstrating how we define prompts to classify the problem (left), reason visual factors (middle), and think about visual observations regarding visual factors (right).
  • Figure 4: Qualitative examples. All the examples are in the case of initial reasonings are not confident. Left: An example in the VCR dataset, where the ITA corrects the initial reasoning. Middle: An example in the A-OKVQA dataset, where the LLM corrects the initial reasoning after giving the observation of the visual factor. Right: An example in the A-OKVQA dataset, where the reasoned clue provides more useful information than VQA.