Enhancing Advanced Visual Reasoning Ability of Large Language Models

TL;DR

This work presents CVR-LLM, a framework that fuses Vision-Language perception with Large Language Model reasoning to tackle complex visual reasoning without extra training. It introduces context-aware image descriptions (CaID), a dual-loop self-refinement process, Complex Visual Reasoning In-Context Learning (CVR-ICL), and the Chain-of-Comparison (CoC) evaluation technique. Across five benchmarks (WinoGAViL, Winoground, Whoops, VCR, NYCCC), CVR-LLM achieves state-of-the-art results, supported by ablations showing the synergy of CaID and CVR-ICL. While offering strong performance and broad applicability, the approach admits higher latency than end-to-end methods and still lags behind GPT-4V on some tasks, guiding future efficiency and accuracy improvements.

Abstract

Recent advancements in Vision-Language (VL) research have sparked new benchmarks for complex visual reasoning, challenging models' advanced reasoning ability. Traditional Vision-Language Models (VLMs) perform well in visual perception tasks while struggling with complex reasoning scenarios. Conversely, Large Language Models (LLMs) demonstrate robust text reasoning capabilities; however, they lack visual acuity. To bridge this gap, we propose Complex Visual Reasoning Large Language Models (CVR-LLM), capitalizing on VLMs' visual perception proficiency and LLMs' extensive reasoning capability. Unlike recent multimodal large language models (MLLMs) that require a projection layer, our approach transforms images into detailed, context-aware descriptions using an iterative self-refinement loop and leverages LLMs' text knowledge for accurate predictions without extra training. We also introduce a novel multi-modal in-context learning (ICL) methodology to enhance LLMs' contextual understanding and reasoning. Additionally, we introduce Chain-of-Comparison (CoC), a step-by-step comparison technique enabling contrasting various aspects of predictions. Our CVR-LLM presents the first comprehensive study across a wide array of complex visual reasoning tasks and achieves SOTA performance among all.

Figures (13)

• Figure 1: Five distinct examples from diverse datasets in the complex visual reasoning field bitton2023breaking challenge AI models' ability of complex reasoning in different aspects such as general commonsense.
• Figure 2: An example of our CVR-LLM works on the Winoground dataset. Our method transfers images into context-aware image descriptions through CaID and leverages the sophisticated reasoning and ICL abilities of LLMs with the CVR-ICL module, offering a more precise answer.
• Figure 3: The framework overview of CaID. It is designed to transfer images into contextualized descriptions, bypassing the need for direct multi-modal fusion and leveraging LLMs' extensive knowledge for more accurate predictions.
• Figure 4: The generic diagram of our proposed CVR-ICL approach. The dual analysis enables our approach to more effectively select contextually relevant examples from text and multi-modal domains.
• Figure 5: Two examples from WinoGAViL compare context-aware image descriptions with general image captions. WinoGAViL is designed to ask the model to select the image that best matches the cue word.