Visually-Augmented Language Modeling

TL;DR

The paper tackles the lack of visual grounding in large-scale pretrained language models by introducing VaLM, a Visually-Augmented Language Model that augments text with retrieved images using a CLIP-based dense retrieval pipeline and a novel visual knowledge fusion layer. By enabling joint attention over textual context and retrieved images, VaLM demonstrates substantial improvements on visual knowledge-intensive commonsense tasks (e.g., MemoryColor, ObjectShape, RelativeSize, PIQA) and yields positive zero-shot gains on standard NLU and LM benchmarks. Key contributions include a flexible on-the-fly text-image alignment mechanism, an image retrieval module with a large cached image knowledge base, and a dedicated fusion layer that effectively integrates visual information into autoregressive pretraining. The results suggest that retrieved visual knowledge can meaningfully reduce hallucinations and enhance grounded language understanding, with practical implications for scalable multimodal pretraining and downstream reasoning tasks.

Abstract

Human language is grounded on multimodal knowledge including visual knowledge like colors, sizes, and shapes. However, current large-scale pre-trained language models rely on text-only self-supervised training with massive text data, which precludes them from utilizing relevant visual information when necessary. To address this, we propose a novel pre-training framework, named VaLM, to Visually-augment text tokens with retrieved relevant images for Language Modeling. Specifically, VaLM builds on a novel latent text-image alignment method via an image retrieval module to fetch corresponding images given a textual context. With the visually-augmented context, VaLM uses a visual knowledge fusion layer to enable multimodal grounded language modeling by attending to both text context and visual knowledge in images. We evaluate VaLM on various visual knowledge-intensive commonsense reasoning tasks, which require visual information to excel. The experimental results illustrate that VaLM outperforms all strong language-only and vision-language baselines with substantial gains in reasoning object commonsense including color, size, and shape. Our code is available at https://github.com/Victorwz/VaLM.

Figures (3)

• Figure 1: Overview of visually-augmented language modeling (VaLM). We conduct dense retrieval to get top-$k$ images for the input context at each time step. Then the visual knowledge fusion layer attends to both text tokens and retrieved images. The vision-language fused representation is fed back to Transformer for language modeling.
• Figure 2: The attention matrix visualization given the query prompt "the color of [object] is" for VaLM. VaLM achieves accurate image retrieval of top-4 images corresponding to the objects of sky and parsley as augmented images, shown in the horizontal index of each subfigure.
• Figure 3: The visualization of the predicted probability distribution on 11 object color types with retrieved images and colorization images, respectively. The adopted prompt for reasoning the object color of an apple is "the color of [object] is".