Improving Visual Commonsense in Language Models via Multiple Image Generation

TL;DR

The paper tackles the mismatch between visual grounding and text-based reasoning in large language models. It introduces vLMIG, a two-component framework that combines a frozen LLM and vision encoder with a Visual Token Projector and a Late Fusion Attention Layer to fuse visual cues with textual context, while also generating multiple images at inference and ensemble-averaging their predictions. Empirical results show strong gains in visual commonsense across object and visual tasks, with additional modest improvements on text-based commonsense reasoning and reading comprehension across model scales. This approach offers a practical path to endow LLMs with multimodal grounding without sacrificing core language abilities, with open-source code available for replication and extension.

Abstract

Commonsense reasoning is fundamentally based on multimodal knowledge. However, existing large language models (LLMs) are primarily trained using textual data only, limiting their ability to incorporate essential visual information. In contrast, Visual Language Models, which excel at visually-oriented tasks, often fail at non-visual tasks such as basic commonsense reasoning. This divergence highlights a critical challenge - the integration of robust visual understanding with foundational text-based language reasoning. To this end, we introduce a method aimed at enhancing LLMs' visual commonsense. Specifically, our method generates multiple images based on the input text prompt and integrates these into the model's decision-making process by mixing their prediction probabilities. To facilitate multimodal grounded language modeling, we employ a late-fusion layer that combines the projected visual features with the output of a pre-trained LLM conditioned on text only. This late-fusion layer enables predictions based on comprehensive image-text knowledge as well as text only when this is required. We evaluate our approach using several visual commonsense reasoning tasks together with traditional NLP tasks, including common sense reasoning and reading comprehension. Our experimental results demonstrate significant superiority over existing baselines. When applied to recent state-of-the-art LLMs (e.g., Llama3), we observe improvements not only in visual common sense but also in traditional NLP benchmarks. Code and models are available under https://github.com/guyyariv/vLMIG.

Figures (3)

• Figure 1: Illustration of the proposed method. During training, we utilize two types of data: (i). a pair of images and the corresponding text description, or (ii) a text and synthetically generated image conditioned on the input text. Each image is passed through a prerained vision encoder and then through a visual token projector, which projects the visual encoding onto pseudo-textual tokens. Simultaneously, the input text is passed through a pre-trained LLM, producing textual tokens. Next, our fusion layer fuses the visual pseudo-textual tokens and textual tokens and produces a prediction of the next textual token. In this fusion layer, an attention-like mechanism is performed where queries are taken to the textual tokens, and the keys and values are taken as both the textual tokens and visual pseudo-textual tokens. In blue are fixed pretrained components while in orange are trainable components. At inference, the same process is applied, but to $k$ different images conditionally generated using the input text. The predictions resulting from different images are then integrated as a form of ensemble using Eq. \ref{['eq:sum_probabilites']} and Eq. \ref{['eq:conf']}.
• Figure 2: An illustrative example of our method at inference. On the LHS, we consider the task of visual commonsense. While Llama3's answer is wrong, our method generates 3 images and places, for the correct class, most weight (score) on the second and third images, thus answering correctly. On the RHS, for text generation, our method generates three images for different parts of the sentence. These images are each used in our method's answer. Llama3's answer, on the other hand, is less visually cohesive.
• Figure 3: Average impact of the number of generated images per inference on performance, aggregating results from three tests: Color xia2023imagenetvc, PIQA bisk2019piqa, and BoolQ clark2019boolq. This graph displays the average performance scores for values of $k$ from 1 to 10, illustrating the general trend across varied test scenarios under identical settings.