Prism: A Framework for Decoupling and Assessing the Capabilities of VLMs

TL;DR

Prism proposes a decoupled framework to separately evaluate perception and reasoning in Vision Language Models by splitting VLM-based visual information extraction (perception) from LLM-based question answering (reasoning). It demonstrates that a lightweight perception module combined with a powerful external LLM can achieve competitive results on multimodal benchmarks like MMStar, while substantially reducing training and deployment costs. The study reveals that proprietary VLMs dominate perception while open-source perception is less sensitive to LLM size, and that small VLMs are often bottlenecked by reasoning capabilities unless aided by external language models. As both an analytical tool and an actionable VLM solver, Prism offers a cost-efficient path to robust vision-language tasks and targeted model improvement across domains.

Abstract

Vision Language Models (VLMs) demonstrate remarkable proficiency in addressing a wide array of visual questions, which requires strong perception and reasoning faculties. Assessing these two competencies independently is crucial for model refinement, despite the inherent difficulty due to the intertwined nature of seeing and reasoning in existing VLMs. To tackle this issue, we present Prism, an innovative framework designed to disentangle the perception and reasoning processes involved in visual question solving. Prism comprises two distinct stages: a perception stage that utilizes a VLM to extract and articulate visual information in textual form, and a reasoning stage that formulates responses based on the extracted visual information using a Large Language Model (LLM). This modular design enables the systematic comparison and assessment of both proprietary and open-source VLM for their perception and reasoning strengths. Our analytical framework provides several valuable insights, underscoring Prism's potential as a cost-effective solution for vision-language tasks. By combining a streamlined VLM focused on perception with a powerful LLM tailored for reasoning, Prism achieves superior results in general vision-language tasks while substantially cutting down on training and operational expenses. Quantitative evaluations show that Prism, when configured with a vanilla 2B LLaVA and freely accessible GPT-3.5, delivers performance on par with VLMs $10 \times$ larger on the rigorous multimodal benchmark MMStar. The project is released at: https://github.com/SparksJoe/Prism.

Figures (9)

• Figure 1: Prism Framework Architecture. Prism framework takes image-query pairs as input. An instruction (can be query-agnostic or query-aware) and the image are first fed into the VLM to extract visual information. Then, an LLM is used to generate the answer based on the reformatted query which combines the original question and visual information in textual form.
• Figure 2: Comparing End-to-End Performance and Perception Capability on MMStar. We display model accuracies in end-to-end VQA and the Prism perception test with query-specific instructions. Most small-scale (7B, 13B, etc.) VLMs achieve better performance within Prism.
• Figure 3: The Effect of Language Model Size on Perception Ability. We compare visual information extracted from different LLaVA-NeXT models. Left: LLaVA-NeXT (Yi-34B) tells the spatial arrangement in a more detailed way; Right: LLaVA-NeXT (Vicuna-7B) dismisses the query on the man's hair while LLaVA-NeXT (Yi-34B) tells all contents elaborately following the instruction.
• Figure 4: Different Generic Instructions we adopted in the Ablation Study.
• Figure 5: The Performance Changes of Using an External LLM (ChatGPT) for Reasoning of Small Scale VLMs.