Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

LLaVA-Gemma: Accelerating Multimodal Foundation Models with a Compact Language Model

Musashi Hinck, Matthew L. Olson, David Cobbley, Shao-Yen Tseng, Vasudev Lal

TL;DR

LLaVA-Gemma investigates accelerating multimodal foundation models by pairing Gemma LLMs (2B and 7B) with vision encoders in a LLaVA-style framework. Through a two-stage training regime and extensive ablations across vision backbones, pretraining, and LM size, the study reveals nuanced, dataset-dependent effects: DinoV2 often helps 2B configurations, pretraining generally aids performance, and larger LM size yields mixed results with notable task-specific gains (e.g., ScienceQA). The results highlight the trade-offs between computation and multimodal understanding in small-scale models and provide public training recipes, code, and weights to enable further exploration. The work contributes practical insights for designing efficient vision-language systems and offers a benchmarked baseline for future research in compact MMFMs.

Abstract

We train a suite of multimodal foundation models (MMFM) using the popular LLaVA framework with the recently released Gemma family of large language models (LLMs). Of particular interest is the 2B parameter Gemma model, which provides opportunities to construct capable small-scale MMFMs. In line with findings from other papers in this space, we test the effect of ablating three design features: pretraining the connector, utilizing a more powerful image backbone, and increasing the size of the language backbone. The resulting models, which we call LLaVA-Gemma, exhibit moderate performance on an array of evaluations, but fail to improve past the current comparably sized SOTA models. Closer analysis of performance shows mixed effects; skipping pretraining tends to reduce performance, larger vision models sometimes improve performance, and increasing language model size has inconsistent effects. We publicly release training recipes, code and weights for our models for the LLaVA-Gemma models.

LLaVA-Gemma: Accelerating Multimodal Foundation Models with a Compact Language Model

TL;DR

LLaVA-Gemma investigates accelerating multimodal foundation models by pairing Gemma LLMs (2B and 7B) with vision encoders in a LLaVA-style framework. Through a two-stage training regime and extensive ablations across vision backbones, pretraining, and LM size, the study reveals nuanced, dataset-dependent effects: DinoV2 often helps 2B configurations, pretraining generally aids performance, and larger LM size yields mixed results with notable task-specific gains (e.g., ScienceQA). The results highlight the trade-offs between computation and multimodal understanding in small-scale models and provide public training recipes, code, and weights to enable further exploration. The work contributes practical insights for designing efficient vision-language systems and offers a benchmarked baseline for future research in compact MMFMs.

Abstract

We train a suite of multimodal foundation models (MMFM) using the popular LLaVA framework with the recently released Gemma family of large language models (LLMs). Of particular interest is the 2B parameter Gemma model, which provides opportunities to construct capable small-scale MMFMs. In line with findings from other papers in this space, we test the effect of ablating three design features: pretraining the connector, utilizing a more powerful image backbone, and increasing the size of the language backbone. The resulting models, which we call LLaVA-Gemma, exhibit moderate performance on an array of evaluations, but fail to improve past the current comparably sized SOTA models. Closer analysis of performance shows mixed effects; skipping pretraining tends to reduce performance, larger vision models sometimes improve performance, and increasing language model size has inconsistent effects. We publicly release training recipes, code and weights for our models for the LLaVA-Gemma models.
Paper Structure (11 sections, 2 figures, 1 table)

This paper contains 11 sections, 2 figures, 1 table.

Table of Contents

  1. Introduction
  2. Methods
  3. Results
  4. Influence of Vision Encoder on Performance
  5. Effects of Pretraining
  6. Comparison to Baselines
  7. Speed of Training and Inference
  8. Analysis
  9. Impact of Alternate Design Choices
  10. Visualizing Attention with Relevancy Maps
  11. Discussion

Figures (2)

  • Figure 1: Effect of design choices differs between evaluations. Point indicates average change in probability of correct answer versus baseline design.
  • Figure 2: Relevancy map comparison between LLaVA-Gemma 2b (Left) and LLaVA-Gemma 7b (Right) with gradients on the first relevant output token. For the question "Is the duck floating? (a) Yes (b) No", despite using the identical CLIP vision encoder, the smaller model does not attend to the visual input.