Multimodal Instruction Tuning with Hybrid State Space Models

TL;DR

This multimodal model can effectively process long context input exceeding 100k tokens, outperforming existing models across various benchmarks, and is the first to be trained on low-resolution images or low-frame-rate videos while being capable of inference on high-resolution images and high-frame-rate videos, offering flexibility for inference in diverse scenarios.

Abstract

Handling lengthy context is crucial for enhancing the recognition and understanding capabilities of multimodal large language models (MLLMs) in applications such as processing high-resolution images or high frame rate videos. The rise in image resolution and frame rate substantially increases computational demands due to the increased number of input tokens. This challenge is further exacerbated by the quadratic complexity with respect to sequence length of the self-attention mechanism. Most prior works either pre-train models with long contexts, overlooking the efficiency problem, or attempt to reduce the context length via downsampling (e.g., identify the key image patches or frames) to decrease the context length, which may result in information loss. To circumvent this issue while keeping the remarkable effectiveness of MLLMs, we propose a novel approach using a hybrid transformer-MAMBA model to efficiently handle long contexts in multimodal applications. Our multimodal model can effectively process long context input exceeding 100k tokens, outperforming existing models across various benchmarks. Remarkably, our model enhances inference efficiency for high-resolution images and high-frame-rate videos by about 4 times compared to current models, with efficiency gains increasing as image resolution or video frames rise. Furthermore, our model is the first to be trained on low-resolution images or low-frame-rate videos while being capable of inference on high-resolution images and high-frame-rate videos, offering flexibility for inference in diverse scenarios.

Figures (4)

• Figure 1: Overview of $\textsc{MMJamba}$. Image representations of all divided patches are flattened and concatenated. The hybrid state space model consists of interleaved Mamba and Transformer layers.
• Figure 2: Efficiency analysis across different models and resolutions. First inference latency is the time used to generate 1 token and throughput is calculated as average tokens processed when generating 1024 tokens.
• Figure 3: Dialogues between the user and MLLMs. Examples on videos are presented in Sec \ref{['sec:video_case']}.
• Figure 4: Qualitative Video Understanding Cases from different perspectives of MMJamba.