Self-Imagine: Effective Unimodal Reasoning with Multimodal Models using Self-Imagination

TL;DR

Self-Imagine introduces a training-free, image-augmented reasoning framework for Vision-Language Models by having a single VLM generate an HTML diagram from a text question, render that HTML as an image, and re-answer the question using both the text and the image. The method relies on few-shot prompts to map questions to HTML representations (h_t = $\textsc{vlm}(p \, || \, q_t, I_d)$) and then uses an image-rendering step (I_g) via an HTML renderer, enabling unimodal tasks like math word problems and general reasoning to benefit from structured visual context. Evaluated on three math datasets ($gsm8k$, $asdiv$, $svamp$) and nine BIG-Bench Hard symbolic tasks with LLAVA-1.5 and Gemini Pro, Self-Imagine yields average improvements of around $+3.1\%$, $+3.2\%$, and $+6.9\%$ on GSM8K, ASDIV, and SVAMP respectively, and substantial gains on several symbolic tasks, though it can hurt some others when HTML or visual representations fail to capture essential information. The results underscore a strong dependency on the quality of the generated image and reveal both task- and model-dependent benefits, motivating future work on more faithful visual representations and broader applicability across reasoning domains.

Abstract

The potential of Vision-Language Models (VLMs) often remains underutilized in handling complex text-based problems, particularly when these problems could benefit from visual representation. Resonating with humans' ability to solve complex text-based problems by (1) creating a visual diagram from the problem and (2) deducing what steps they need to take to solve it, we propose Self-Imagine. We leverage a single Vision-Language Model (VLM) to generate a structured representation of the question using HTML, then render the HTML as an image, and finally use the same VLM to answer the question using both the question and the image. Our approach does not require any additional training data or training. We evaluate our approach on three mathematics tasks and nine general-purpose reasoning tasks using state-of-the-art (LLAVA-1.5 and GEMINI PRO) VLMs. Our approach boosts the performance of LLAVA-1.5 and GEMINI PRO on all math tasks (on average GSM8K: +3.1%; ASDIV: +3.2%; SVAMP: +6.9%) and the majority of the general-purpose reasoning tasks by 3.2% to 6.0% on average.

Figures (9)

• Figure 1: Generating an image from a question via a single VLM through HTML.
• Figure 2: [Left] Reasoning using vlm without Self-Imagine: Given a question (0), the vlm generates an answer (1). [Right] Reasoning using vlm with Self-Imagine: Given a question (0), the vlm generates a structured representation of the question using HTML (1). The HTML is rendered as an image (2) which is then passed along with the question to the vlm again (3). The vlm finally generates the answer by combining both vision and language modalities (4).
• Figure 3: Self-Imagine main results: Self-Imagine improves accuracy over a diverse range of mathematical and symbolic reasoning tasks.
• Figure 4: Example from math world problem tasks.
• Figure 5: Examples from some BIG-Bench Hard tasks.