Mobile Foundation Model as Firmware
Jinliang Yuan, Chen Yang, Dongqi Cai, Shihe Wang, Xin Yuan, Zeling Zhang, Xiang Li, Dingge Zhang, Hanzi Mei, Xianqing Jia, Shangguang Wang, Mengwei Xu
TL;DR
The paper presents a vision and prototype for a mobile foundation model (M4) that is treated as firmware, co-managed by the mobile OS and hardware to serve a wide range of on-device AI tasks. M4 uses a three-component architecture (Multimodal Embedding, Foundation Backbone, Multimodal Generator) and lightweight PEFT adapters to tailor performance per task, enabling cross-task sharing and reduced hardware design complexity. It is evaluated on a novel edge-oriented benchmark, eAIBench, across 38 tasks and 50 datasets, showing parity with task-specific models on about $85\%$ of tasks and demonstrating strong zero-/few-shot capabilities and scalability advantages. While current runtimes on mobile hardware are slower than task-specific models, the architecture reduces operator diversity and paves the way for future NPUs to realize competitive latency and energy profiles; the approach promises simpler hardware design and OS-wide AI scheduling, with practical multimodal applications demonstrated. Overall, M4 represents an initial but promising step toward a firmware-like mobile foundation model that can unify and accelerate on-device AI across apps and modalities.
Abstract
In today's landscape, smartphones have evolved into hubs for hosting a multitude of deep learning models aimed at local execution. A key realization driving this work is the notable fragmentation among these models, characterized by varied architectures, operators, and implementations. This fragmentation imposes a significant burden on the comprehensive optimization of hardware, system settings, and algorithms. Buoyed by the recent strides in large foundation models, this work introduces a pioneering paradigm for mobile AI: a collaborative management approach between the mobile OS and hardware, overseeing a foundational model capable of serving a broad spectrum of mobile AI tasks, if not all. This foundational model resides within the NPU and remains impervious to app or OS revisions, akin to firmware. Concurrently, each app contributes a concise, offline fine-tuned "adapter" tailored to distinct downstream tasks. From this concept emerges a concrete instantiation known as \sys. It amalgamates a curated selection of publicly available Large Language Models (LLMs) and facilitates dynamic data flow. This concept's viability is substantiated through the creation of an exhaustive benchmark encompassing 38 mobile AI tasks spanning 50 datasets, including domains such as Computer Vision (CV), Natural Language Processing (NLP), audio, sensing, and multimodal inputs. Spanning this benchmark, \sys unveils its impressive performance. It attains accuracy parity in 85\% of tasks, demonstrates improved scalability in terms of storage and memory, and offers satisfactory inference speed on Commercial Off-The-Shelf (COTS) mobile devices fortified with NPU support. This stands in stark contrast to task-specific models tailored for individual applications.