Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

OpenBA-V2: Reaching 77.3% High Compression Ratio with Fast Multi-Stage Pruning

Dan Qiao, Yi Su, Pinzheng Wang, Jing Ye, Wenjing Xie, Yuechi Zhou, Yuyang Ding, Zecheng Tang, Jikai Wang, Yixin Ji, Yue Wang, Pei Guo, Zechen Sun, Zikang Zhang, Juntao Li, Pingfu Chao, Wenliang Chen, Guohong Fu, Guodong Zhou, Qiaoming Zhu, Min Zhang

TL;DR

OpenBA-V2 targets efficient deployment of LLMs by compressing a 15B model to 3.4B through a multi-stage pruning pipeline and continual pre-training. It combines layer, neural, and vocabulary pruning with UL2-based training variants, including Dynamic-UL2 and Mix-Denoising to maintain performance while boosting training efficiency. The authors also expand data and instruction resources via BiFlan-V2, enabling a data-rich, instruction-tuned compact model that approaches the capabilities of larger models on several tasks and excels in NER fine-tuning. Overall, the work demonstrates that careful pruning, data strategies, and training objectives can yield high compression with minimal loss, enabling practical deployment in resource-constrained settings.

Abstract

Large Language Models (LLMs) have played an important role in many fields due to their powerful capabilities.However, their massive number of parameters leads to high deployment requirements and incurs significant inference costs, which impedes their practical applications. Training smaller models is an effective way to address this problem. Therefore, we introduce OpenBA-V2, a 3.4B model derived from multi-stage compression and continual pre-training from the original 15B OpenBA model. OpenBA-V2 utilizes more data, more flexible training objectives, and techniques such as layer pruning, neural pruning, and vocabulary pruning to achieve a compression rate of 77.3\% with minimal performance loss. OpenBA-V2 demonstrates competitive performance compared to other open-source models of similar size, achieving results close to or on par with the 15B OpenBA model in downstream tasks such as common sense reasoning and Named Entity Recognition (NER). OpenBA-V2 illustrates that LLMs can be compressed into smaller ones with minimal performance loss by employing advanced training objectives and data strategies, which may help deploy LLMs in resource-limited scenarios.

OpenBA-V2: Reaching 77.3% High Compression Ratio with Fast Multi-Stage Pruning

TL;DR

OpenBA-V2 targets efficient deployment of LLMs by compressing a 15B model to 3.4B through a multi-stage pruning pipeline and continual pre-training. It combines layer, neural, and vocabulary pruning with UL2-based training variants, including Dynamic-UL2 and Mix-Denoising to maintain performance while boosting training efficiency. The authors also expand data and instruction resources via BiFlan-V2, enabling a data-rich, instruction-tuned compact model that approaches the capabilities of larger models on several tasks and excels in NER fine-tuning. Overall, the work demonstrates that careful pruning, data strategies, and training objectives can yield high compression with minimal loss, enabling practical deployment in resource-constrained settings.

Abstract

Large Language Models (LLMs) have played an important role in many fields due to their powerful capabilities.However, their massive number of parameters leads to high deployment requirements and incurs significant inference costs, which impedes their practical applications. Training smaller models is an effective way to address this problem. Therefore, we introduce OpenBA-V2, a 3.4B model derived from multi-stage compression and continual pre-training from the original 15B OpenBA model. OpenBA-V2 utilizes more data, more flexible training objectives, and techniques such as layer pruning, neural pruning, and vocabulary pruning to achieve a compression rate of 77.3\% with minimal performance loss. OpenBA-V2 demonstrates competitive performance compared to other open-source models of similar size, achieving results close to or on par with the 15B OpenBA model in downstream tasks such as common sense reasoning and Named Entity Recognition (NER). OpenBA-V2 illustrates that LLMs can be compressed into smaller ones with minimal performance loss by employing advanced training objectives and data strategies, which may help deploy LLMs in resource-limited scenarios.
Paper Structure (37 sections, 8 figures, 7 tables, 1 algorithm)

This paper contains 37 sections, 8 figures, 7 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Related Work
  3. Lightweight LLMs Pre-trained from Scratch
  4. Model Compression for LLMs
  5. Dataset Preparation
  6. Pre-training Data Collection and Processing
  7. Data Sources
  8. Data Processing
  9. Data Statistics
  10. BiFlan-V2: Instruction Data Collection
  11. English Instruction Data Collection
  12. Chinese Instruction Data Collection
  13. BiFlan-V2 Data Statistics
  14. Fast Multi-Stage Pruning
  15. Pruning Strategies
  16. ...and 22 more sections

Figures (8)

  • Figure 1: Data distribution of different training corpus, where Figure (a) shows the distribution of the pre-training data and Figure (b) shows the distribution of the BiFlan-V2 instruction data.
  • Figure 2: Illustraion of random neural pruning.
  • Figure 3: Illustraion of how to prune the vocabulary and the corresponding embedding.
  • Figure 4: Illustration of Optimized-UL2, which allows training with various types of noise with few padding tokens.
  • Figure 5: Model performance with different pruning strategies and pruning parameters.
  • ...and 3 more figures