SOAEsV2-7B/72B: Full-Pipeline Optimization for State-Owned Enterprise LLMs via Continual Pre-Training, Domain-Progressive SFT and Distillation-Enhanced Speculative Decoding

TL;DR

The paper addresses the need for high-capacity, domain-aware LLMs in the SOAEs sector where general models struggle to balance domain knowledge with broad language skills. It presents SOAEsV2-7B/72B, a three-phase full-pipeline comprising continual pre-training on a $72$B base, domain-progressive SFT, and distillation-enhanced speculative decoding to accelerate inference. Results indicate the $72$B model retains $99.8\%$ of base general capabilities while achieving domain gains (Rouge-1 by $1.08\times$, BLEU-4 by $1.17\times$), and inference speedups of $1.39\times$ to $1.52\times$ via SPD; ablations confirm the superiority of domain-progressive SFT over single-stage training. The work provides a scalable, transferable blueprint for deploying domain-specific LLMs beyond SOAEs, with potential extension to finance, healthcare, and other knowledge-intensive domains.

Abstract

This study addresses key challenges in developing domain-specific large language models (LLMs) for Chinese state-owned assets and enterprises (SOAEs), where current approaches face three limitations: 1) constrained model capacity that limits knowledge integration and cross-task adaptability; 2) excessive reliance on domain-specific supervised fine-tuning (SFT) data, which neglects the broader applicability of general language patterns; and 3) inefficient inference acceleration for large models processing long contexts. In this work, we propose SOAEsV2-7B/72B, a specialized LLM series developed via a three-phase framework: 1) continual pre-training integrates domain knowledge while retaining base capabilities; 2) domain-progressive SFT employs curriculum-based learning strategy, transitioning from weakly relevant conversational data to expert-annotated SOAEs datasets to optimize domain-specific tasks; 3) distillation-enhanced speculative decoding accelerates inference via logit distillation between 72B target and 7B draft models, achieving 1.39-1.52$\times$ speedup without quality loss. Experimental results demonstrate that our domain-specific pre-training phase maintains 99.8% of original general language capabilities while significantly improving domain performance, resulting in a 1.08$\times$ improvement in Rouge-1 score and a 1.17$\times$ enhancement in BLEU-4 score. Ablation studies further show that domain-progressive SFT outperforms single-stage training, achieving 1.02$\times$ improvement in Rouge-1 and 1.06$\times$ in BLEU-4. Our work introduces a comprehensive, full-pipeline approach for optimizing SOAEs LLMs, bridging the gap between general language capabilities and domain-specific expertise.

Figures (1)

• Figure 1: Pipeline of our document layout analysis algorithm