Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Improving the Efficiency of Visually Augmented Language Models

Paula Ontalvilla, Aitor Ormazabal, Gorka Azkune

TL;DR

This work challenges the necessity of explicit image retrieval for visually augmenting autoregressive LMs by introducing Blind-VaLM, which integrates visually grounded textual representations from CLIP directly into the fusion layer. Blind-VaLM matches VaLM on Visual Language Understanding, Natural Language Understanding, and language modeling tasks while requiring significantly less computation, and can outperform VaLM when scaled within the same compute budget. The approach demonstrates substantial efficiency gains and suggests that textual grounding alone can capture the essential visual properties needed for augmentation. These findings open avenues for more cost-effective multimodal augmentation and broader exploration of multimodal encoders beyond image retrieval.

Abstract

Despite the impressive performance of autoregressive Language Models (LM) it has been shown that due to reporting bias, LMs lack visual knowledge, i.e. they do not know much about the visual world and its properties. To augment LMs with visual knowledge, existing solutions often rely on explicit images, requiring time-consuming retrieval or image generation systems. This paper shows that explicit images are not necessary to visually augment an LM. Instead, we use visually-grounded text representations obtained from the well-known CLIP multimodal system. For a fair comparison, we modify VALM, a visually-augmented LM which uses image retrieval and representation, to work directly with visually-grounded text representations. We name this new model BLIND-VALM. We show that BLIND-VALM performs on par with VALM for Visual Language Understanding (VLU), Natural Language Understanding (NLU) and Language Modeling tasks, despite being significantly more efficient and simpler. We also show that scaling up our model within the compute budget of VALM, either increasing the model or pre-training corpus size, we outperform VALM for all the evaluation tasks.

Improving the Efficiency of Visually Augmented Language Models

TL;DR

This work challenges the necessity of explicit image retrieval for visually augmenting autoregressive LMs by introducing Blind-VaLM, which integrates visually grounded textual representations from CLIP directly into the fusion layer. Blind-VaLM matches VaLM on Visual Language Understanding, Natural Language Understanding, and language modeling tasks while requiring significantly less computation, and can outperform VaLM when scaled within the same compute budget. The approach demonstrates substantial efficiency gains and suggests that textual grounding alone can capture the essential visual properties needed for augmentation. These findings open avenues for more cost-effective multimodal augmentation and broader exploration of multimodal encoders beyond image retrieval.

Abstract

Despite the impressive performance of autoregressive Language Models (LM) it has been shown that due to reporting bias, LMs lack visual knowledge, i.e. they do not know much about the visual world and its properties. To augment LMs with visual knowledge, existing solutions often rely on explicit images, requiring time-consuming retrieval or image generation systems. This paper shows that explicit images are not necessary to visually augment an LM. Instead, we use visually-grounded text representations obtained from the well-known CLIP multimodal system. For a fair comparison, we modify VALM, a visually-augmented LM which uses image retrieval and representation, to work directly with visually-grounded text representations. We name this new model BLIND-VALM. We show that BLIND-VALM performs on par with VALM for Visual Language Understanding (VLU), Natural Language Understanding (NLU) and Language Modeling tasks, despite being significantly more efficient and simpler. We also show that scaling up our model within the compute budget of VALM, either increasing the model or pre-training corpus size, we outperform VALM for all the evaluation tasks.
Paper Structure (18 sections, 1 figure, 6 tables)

This paper contains 18 sections, 1 figure, 6 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Blind-VaLM architecture
  4. Experimental setup
  5. Text Corpus.
  6. Image data and retrieval module.
  7. Pre-training hyperparameters.
  8. Evaluation.
  9. Scaling Blind-VaLM.
  10. (i) Scaling model size.
  11. (ii) Scaling pre-training compute.
  12. Results
  13. Blind-VaLM matches VaLM on VLU, NLU and LM tasks.
  14. Blind-VaLM outperforms VaLM, when trained within the same compute budget.
  15. Conclusions
  16. ...and 3 more sections

Figures (1)

  • Figure 1: Architecture comparison of the original VaLM (left) and our proposed Blind-VaLM (right).