ixi-GEN: Efficient Industrial sLLMs through Domain Adaptive Continual Pretraining

TL;DR

ixi-GEN introduces Domain Adaptive Continual Pretraining (DACP) to efficiently adapt small LLMs to industrial domains while preserving general capabilities. By combining mid-scale domain data with a pseudo-replay dataset and applying post-training instruction tuning, DACP delivers domain gains across Telco and Finance with reduced infrastructure needs. Extensive benchmarks and real-world evaluations show that DACP-applied sLLMs can outperform larger general-domain counterparts in target domains, enabling cost-effective enterprise deployment without sacrificing user experience. The approach is demonstrated to be robust across backbone models and domains, offering a practical path for scalable, domain-aware sLLMs in industry.

Abstract

The emergence of open-source large language models (LLMs) has expanded opportunities for enterprise applications; however, many organizations still lack the infrastructure to deploy and maintain large-scale models. As a result, small LLMs (sLLMs) have become a practical alternative despite inherent performance limitations. While Domain Adaptive Continual Pretraining (DACP) has been explored for domain adaptation, its utility in commercial settings remains under-examined. In this study, we validate the effectiveness of a DACP-based recipe across diverse foundation models and service domains, producing DACP-applied sLLMs (ixi-GEN). Through extensive experiments and real-world evaluations, we demonstrate that ixi-GEN models achieve substantial gains in target-domain performance while preserving general capabilities, offering a cost-efficient and scalable solution for enterprise-level deployment.

Figures (6)

• Figure 1: DACP-applied sLLM outperforms the base model in both the Telco domain and commercial service tasks, while preserving general domain capabilities, thereby enabling cost-efficient yet high-performance service delivery.
• Figure 2: Workflow for domain-adapted sLLM development. The process begins with domain-adaptive continual pretraining (DACP) using both general and domain corpora, followed by instruction tuning to produce domain adapted models. These models are exposed via APIs and optionally fine-tuned, enabling commercial service applications. Evaluation and feedback loops ensure continuous improvement.
• Figure 3: Model performance on general and Telco benchmarks with varying proportions of replay and Telco dataset proportions.
• Figure 4: Screenshot and dialogue example of NW QA system. Industrial sLLM usage requires balancing domain adaptation with general knowledge, as overfitting to a domain can harm overall user experience. The translated version is provided in Table \ref{['tab:figure3_conversation_translation']} in Appendix.
• Figure 5: Success / fail rates of telco-adapted vs. base QA models after service SFT, highlighting the gain from Telco adaptation.