Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Meteor: Mamba-based Traversal of Rationale for Large Language and Vision Models

Byung-Kwan Lee, Chae Won Kim, Beomchan Park, Yong Man Ro

TL;DR

A new efficient LLVM, Mamba-based traversal of rationales (Meteor), which leverages multifaceted rationale to enhance understanding and answering capabilities, achieves significant improvements in vision language performances across multiple evaluation benchmarks requiring diverse capabilities.

Abstract

The rapid development of large language and vision models (LLVMs) has been driven by advances in visual instruction tuning. Recently, open-source LLVMs have curated high-quality visual instruction tuning datasets and utilized additional vision encoders or multiple computer vision models in order to narrow the performance gap with powerful closed-source LLVMs. These advancements are attributed to multifaceted information required for diverse capabilities, including fundamental image understanding, real-world knowledge about common-sense and non-object concepts (e.g., charts, diagrams, symbols, signs, and math problems), and step-by-step procedures for solving complex questions. Drawing from the multifaceted information, we present a new efficient LLVM, Mamba-based traversal of rationales (Meteor), which leverages multifaceted rationale to enhance understanding and answering capabilities. To embed lengthy rationales containing abundant information, we employ the Mamba architecture, capable of processing sequential data with linear time complexity. We introduce a new concept of traversal of rationale that facilitates efficient embedding of rationale. Subsequently, the backbone multimodal language model (MLM) is trained to generate answers with the aid of rationale. Through these steps, Meteor achieves significant improvements in vision language performances across multiple evaluation benchmarks requiring diverse capabilities, without scaling up the model size or employing additional vision encoders and computer vision models.

Meteor: Mamba-based Traversal of Rationale for Large Language and Vision Models

TL;DR

A new efficient LLVM, Mamba-based traversal of rationales (Meteor), which leverages multifaceted rationale to enhance understanding and answering capabilities, achieves significant improvements in vision language performances across multiple evaluation benchmarks requiring diverse capabilities.

Abstract

The rapid development of large language and vision models (LLVMs) has been driven by advances in visual instruction tuning. Recently, open-source LLVMs have curated high-quality visual instruction tuning datasets and utilized additional vision encoders or multiple computer vision models in order to narrow the performance gap with powerful closed-source LLVMs. These advancements are attributed to multifaceted information required for diverse capabilities, including fundamental image understanding, real-world knowledge about common-sense and non-object concepts (e.g., charts, diagrams, symbols, signs, and math problems), and step-by-step procedures for solving complex questions. Drawing from the multifaceted information, we present a new efficient LLVM, Mamba-based traversal of rationales (Meteor), which leverages multifaceted rationale to enhance understanding and answering capabilities. To embed lengthy rationales containing abundant information, we employ the Mamba architecture, capable of processing sequential data with linear time complexity. We introduce a new concept of traversal of rationale that facilitates efficient embedding of rationale. Subsequently, the backbone multimodal language model (MLM) is trained to generate answers with the aid of rationale. Through these steps, Meteor achieves significant improvements in vision language performances across multiple evaluation benchmarks requiring diverse capabilities, without scaling up the model size or employing additional vision encoders and computer vision models.
Paper Structure (22 sections, 28 figures, 18 tables)

This paper contains 22 sections, 28 figures, 18 tables.

Table of Contents

  1. Introduction
  2. Related Works
  3. Rationale-Guided Prediction.
  4. Large Language and Vision Models.
  5. Meteor: Mamba-based traversal of rationale
  6. Model Architecture.
  7. Configuration of Visual Instruction Tuning Dataset.
  8. Curating Rationale.
  9. Mamba Architecture.
  10. Traversal of Rationale.
  11. Training Strategy.
  12. Experiment
  13. Implementation Detail.
  14. Evaluation.
  15. Ablation Studies.
  16. ...and 7 more sections

Figures (28)

  • Figure 1: Across 7B to over 110B parameters, comparing lots of open- and closed-source LLVMs with Meteor on MME fu2023mme, MMB liu2023mmbench, MathVista lu2023mathvista, and AI2D kembhavi2016diagram requiring diverse capabilities for image understanding, common-sense knowledge, non-object concept understanding, etc.
  • Figure 2: Overall comparison of Meteor compared with other open- and closed-source LLVMs.
  • Figure 3: Overview of Meteor architecture and its training steps. Note that, 'Meteor-Multimodal Language Model (MLM)' indicates that as training progresses, the pretrained language model evolves into a multimodal language model.
  • Figure 4: Illuminating how the feature correspondences of cosine similarity are computed under the trained Meteor-Mamba, and showing the feature disparity for <tor> with/without rationale.
  • Figure :
  • ...and 23 more figures