MobileLLM-Pro Technical Report

TL;DR

MobileLLM-Pro targets efficient, high-quality on-device language modeling at 1B parameters, enabling long context up to 128k tokens and practical 4-bit quantization. It introduces four innovations—implicit positional distillation, specialist model merging, simulation-driven data mixing, and quantization-aware training—to achieve SOTA pre-training results across 11 benchmarks and strong instruction-following performance. The training pipeline blends offline data-mix simulation, long-context extension without data-shift, parallel specialist annealing, and QAT for CPU/accelerator deployment, while preserving on-device latency and memory constraints. The work demonstrates robust performance under quantization, provides comprehensive ablations, latency benchmarks, and human evaluations, and releases model weights and code to support future research in efficient on-device LLMs.

Abstract

Efficient on-device language models around 1 billion parameters are essential for powering low-latency AI applications on mobile and wearable devices. However, achieving strong performance in this model class, while supporting long context windows and practical deployment remains a significant challenge. We introduce MobileLLM-Pro, a 1-billion-parameter language model optimized for on-device deployment. MobileLLM-Pro achieves state-of-the-art results across 11 standard benchmarks, significantly outperforming both Gemma 3-1B and Llama 3.2-1B, while supporting context windows of up to 128,000 tokens and showing only minor performance regressions at 4-bit quantization. These improvements are enabled by four core innovations: (1) implicit positional distillation, a novel technique that effectively instills long-context capabilities through knowledge distillation; (2) a specialist model merging framework that fuses multiple domain experts into a compact model without parameter growth; (3) simulation-driven data mixing using utility estimation; and (4) 4-bit quantization-aware training with self-distillation. We release our model weights and code to support future research in efficient on-device language models.

Figures (6)

• Figure 1: Schematic of our four-staged pre-training approach with three performance-specific pre-training phases and an additional quantization stage.
• Figure 2: Schematic of the first two pre-training phases, showing the lack of long context positional embeddings in the student model at the beginning of Phase 2. Training the student using teacher logits, which implicitly contain long-context positional information, we are able to teach the student model to learn long-context relationships
• Figure 3: Exploration of the RoPE angular space using concatenated short-context data (left) and true long-context data (right). Purple=Short-context first stage training, Radius=Number of samples trained at the angle in the polar space, $\alpha$=Maximal rotational distance without the teacher model, limited by the short-context training data.
• Figure 4: Schematic of the Specialist Model Merging approach, showing the initial base model checkpoint being branched out into parallel training trajectories. Each trajectory is learning emergent specializations in isolation, before they are merged into the final, unified model combining the trajectory expertise.
• Figure 5: Overview of our three instruction fine-tuning phases. Similar to pre-training, the instruction training token budget progressively decreases across phases (indicated by the size of blocks). In the second instruction-tuning stage, we use external leave-one-out (LOO) information to better adjust the training data-mix.