PanGu-$π$ Pro:Rethinking Optimization and Architecture for Tiny Language Models

TL;DR

The paper tackles the challenge of deploying high-performance language models on edge devices by conducting a focused empirical study across three axes—neural architecture, parameter initialization, and optimization—for tiny LLMs. It introduces design formulas such as compact tokenizers, depth-oriented architecture tweaks, data-driven parameter inheritance, and multi-round training to boost performance within strict parameter budgets. Empirical results on PanGu-π Pro variants (1B and 1.5B) on a 1.6T multilingual corpus demonstrate notable gains, with 1.5B Pro achieving SOTA performance among similarly sized tiny LLMs and outperforming larger open-source rivals with fewer parameters. The work also provides practical guidelines and release-ready code, offering actionable insights for edge-deployment of LLMs and suggesting avenues for future research in tokenizer learning and hardware-aware architectures.

Abstract

The power of large language models (LLMs) has been demonstrated through numerous data and computing resources. However, the application of language models on mobile devices is facing huge challenge on the computation and memory costs, that is, tiny language models with high performance are urgently required. Limited by the highly complex training process, there are many details for optimizing language models that are seldom studied carefully. In this study, based on a tiny language model with 1B parameters, we carefully design a series of empirical study to analyze the effect of each component. Three perspectives are mainly discussed, \ie, neural architecture, parameter initialization, and optimization strategy. Several design formulas are empirically proved especially effective for tiny language models, including tokenizer compression, architecture tweaking, parameter inheritance and multiple-round training. Then we train PanGu-$π$-1B Pro and PanGu-$π$-1.5B Pro on 1.6T multilingual corpora, following the established formulas. Experimental results demonstrate the improved optimization and architecture yield a notable average improvement of 8.87 on benchmark evaluation sets for PanGu-$π$-1B Pro. Besides, PanGu-$π$-1.5B Pro surpasses a range of SOTA models with larger model sizes, validating its superior performance. The code is available at https://github.com/YuchuanTian/RethinkTinyLM.

Figures (12)

• Figure 1: PanGu-$\pi$ Pro with improved architecture and optimization methods. PanGu-$\pi$-1B wang2023PanGu directly use the developing strategies of LLMs while PanGu-$\pi$-1B Pro achieves an average performance improvement of 8.87 with our methodology. It is worth mentioning that PanGu-$\pi$-1.5B Pro outperforms Qwen-1.8B qwen with 16.67% fewer parameters.
• Figure 2: Accumulative frequency of the top-k vocabularies, where 97.86% data can be represented by a small 48k tokenizer.
• Figure 3: The parameter proportions of model's main body and tokenizer. (a) The large tokenizer inherited from large multilingual models wang2023PanGu. (b) Compact tokenizer by removing low-frequency vocabularies.
• Figure 4: Performance varies w.r.t. model's width, depth and expansion rate. The experiments are conducted on a streamlined dataset comprising 5B tokens. The accuracy is averaged among ARC Easy, HellaSwag and C3. Spearman coefficient is used to measure the correlation between performance and model's configure.
• Figure 5: Performance of large language models when skipping a few layers. "$x$ Skip" denotes adjacent $x$ layers are discarded. Redundancies are observed within intermediate layers while the layers situated near the beginning and end are crucial for maintaining performance.