Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Imp: Highly Capable Large Multimodal Models for Mobile Devices

Zhenwei Shao, Zhou Yu, Jun Yu, Xuecheng Ouyang, Lihao Zheng, Zhenbiao Gai, Mingyang Wang, Jiajun Ding

TL;DR

This work tackles the practical limits of deploying multimodal large models on resource-constrained devices by conducting a systematic study of lightweight LMMs across architecture, training strategies, and data. Building from LLaVA-1.5, the authors derive the Imp family (2B–4B) and show that Imp-3B achieves state-of-the-art performance among lightweight LMMs and even rivals 13B-scale models, while enabling on-device inference with low-bit quantization on mobile GPUs. Key innovations include a Phi-2-based lightweight LLM, SigLIP visual encoders, LoRA-based fine-tuning with rank 256, 2-epoch training, and augmented stage-2 data consisting of OCR/chart and GPT4V-annotated sources, collectively yielding strong multimodal capabilities. They release Imp models and build ImpChat, an offline web/mobile assistant platform, demonstrating practical on-device LMM deployment and a roadmap for future enhancements such as data expansion, distillation, and broader modalities. This work thus provides actionable baselines and a path toward robust, privacy-preserving multimodal AI for edge devices.

Abstract

By harnessing the capabilities of large language models (LLMs), recent large multimodal models (LMMs) have shown remarkable versatility in open-world multimodal understanding. Nevertheless, they are usually parameter-heavy and computation-intensive, thus hindering their applicability in resource-constrained scenarios. To this end, several lightweight LMMs have been proposed successively to maximize the capabilities under constrained scale (e.g., 3B). Despite the encouraging results achieved by these methods, most of them only focus on one or two aspects of the design space, and the key design choices that influence model capability have not yet been thoroughly investigated. In this paper, we conduct a systematic study for lightweight LMMs from the aspects of model architecture, training strategy, and training data. Based on our findings, we obtain Imp -- a family of highly capable LMMs at the 2B-4B scales. Notably, our Imp-3B model steadily outperforms all the existing lightweight LMMs of similar size, and even surpasses the state-of-the-art LMMs at the 13B scale. With low-bit quantization and resolution reduction techniques, our Imp model can be deployed on a Qualcomm Snapdragon 8Gen3 mobile chip with a high inference speed of about 13 tokens/s.

Imp: Highly Capable Large Multimodal Models for Mobile Devices

TL;DR

This work tackles the practical limits of deploying multimodal large models on resource-constrained devices by conducting a systematic study of lightweight LMMs across architecture, training strategies, and data. Building from LLaVA-1.5, the authors derive the Imp family (2B–4B) and show that Imp-3B achieves state-of-the-art performance among lightweight LMMs and even rivals 13B-scale models, while enabling on-device inference with low-bit quantization on mobile GPUs. Key innovations include a Phi-2-based lightweight LLM, SigLIP visual encoders, LoRA-based fine-tuning with rank 256, 2-epoch training, and augmented stage-2 data consisting of OCR/chart and GPT4V-annotated sources, collectively yielding strong multimodal capabilities. They release Imp models and build ImpChat, an offline web/mobile assistant platform, demonstrating practical on-device LMM deployment and a roadmap for future enhancements such as data expansion, distillation, and broader modalities. This work thus provides actionable baselines and a path toward robust, privacy-preserving multimodal AI for edge devices.

Abstract

By harnessing the capabilities of large language models (LLMs), recent large multimodal models (LMMs) have shown remarkable versatility in open-world multimodal understanding. Nevertheless, they are usually parameter-heavy and computation-intensive, thus hindering their applicability in resource-constrained scenarios. To this end, several lightweight LMMs have been proposed successively to maximize the capabilities under constrained scale (e.g., 3B). Despite the encouraging results achieved by these methods, most of them only focus on one or two aspects of the design space, and the key design choices that influence model capability have not yet been thoroughly investigated. In this paper, we conduct a systematic study for lightweight LMMs from the aspects of model architecture, training strategy, and training data. Based on our findings, we obtain Imp -- a family of highly capable LMMs at the 2B-4B scales. Notably, our Imp-3B model steadily outperforms all the existing lightweight LMMs of similar size, and even surpasses the state-of-the-art LMMs at the 13B scale. With low-bit quantization and resolution reduction techniques, our Imp model can be deployed on a Qualcomm Snapdragon 8Gen3 mobile chip with a high inference speed of about 13 tokens/s.
Paper Structure (24 sections, 1 equation, 6 figures, 4 tables)

This paper contains 24 sections, 1 equation, 6 figures, 4 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Large language models (LLMs).
  4. Large multimodal models (LMMs).
  5. Lightweight LLMs and LMMs.
  6. Preliminaries
  7. Model architecture.
  8. Training recipe.
  9. A Roadmap from LLaVA to Imp
  10. Optimized Model Architectures
  11. Choice of the LLM.
  12. Choice of the visual encoder.
  13. Improved Training Strategies
  14. Finetuning mechanism.
  15. Number of training epochs.
  16. ...and 9 more sections

Figures (6)

  • Figure 1: LLaVA-1.5's model architecture and its two-stage training recipe.
  • Figure 2: An overview roadmap from LLaVA-1.5-7B to Imp-3B. The average score is calculated on six commonly-used LMM benchmarks, namely VQAv2 goyal2017vqav2, GQA hudson2019gqa, TextVQA singh2019textvqa, ScienceQA-IMG lu2022learn, POPE li2023pope, and MMB (dev) liu2023mmbench. More detailed results can be referred to Table \ref{['tab:ablations']}.
  • Figure 3: Comprehensive skill demonstrations of Imp, including code generation, math problem solving, Chinese conversation, and medical image understanding.
  • Figure 4: More diverse skill demonstrations of Imp, including camouflage perception, celebrity recognition, chart understanding, and object grounding.
  • Figure 5: Failure cases of Imp, including the hard examples about dense object counting, and long-text OCR.
  • ...and 1 more figures