Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Steel-LLM:From Scratch to Open Source -- A Personal Journey in Building a Chinese-Centric LLM

Qingshui Gu, Shu Li, Tianyu Zheng, Zhaoxiang Zhang

TL;DR

Steel-LLM demonstrates that a compact, open-source Chinese-centric LLM can achieve competitive benchmarks under restricted compute by combining Soft MoE in the FFN with an enhanced FFN, a resourceful training framework, and careful data strategy. The work provides full transparency, including the training pipeline, datasets, intermediate checkpoints, and ablation results, to facilitate reproducibility. Key contributions include a detailed resource-efficient architecture, an efficient pretraining and post-training workflow, and evidence that data distribution and exam-style tasks improve performance on Chinese and multilingual benchmarks. The open-source release and practical guidance have substantial implications for accessible LLM development among smaller research teams and non-English language communities.

Abstract

Steel-LLM is a Chinese-centric language model developed from scratch with the goal of creating a high-quality, open-source model despite limited computational resources. Launched in March 2024, the project aimed to train a 1-billion-parameter model on a large-scale dataset, prioritizing transparency and the sharing of practical insights to assist others in the community. The training process primarily focused on Chinese data, with a small proportion of English data included, addressing gaps in existing open-source LLMs by providing a more detailed and practical account of the model-building journey. Steel-LLM has demonstrated competitive performance on benchmarks such as CEVAL and CMMLU, outperforming early models from larger institutions. This paper provides a comprehensive summary of the project's key contributions, including data collection, model design, training methodologies, and the challenges encountered along the way, offering a valuable resource for researchers and practitioners looking to develop their own LLMs. The model checkpoints and training script are available at https://github.com/zhanshijinwat/Steel-LLM.

Steel-LLM:From Scratch to Open Source -- A Personal Journey in Building a Chinese-Centric LLM

TL;DR

Steel-LLM demonstrates that a compact, open-source Chinese-centric LLM can achieve competitive benchmarks under restricted compute by combining Soft MoE in the FFN with an enhanced FFN, a resourceful training framework, and careful data strategy. The work provides full transparency, including the training pipeline, datasets, intermediate checkpoints, and ablation results, to facilitate reproducibility. Key contributions include a detailed resource-efficient architecture, an efficient pretraining and post-training workflow, and evidence that data distribution and exam-style tasks improve performance on Chinese and multilingual benchmarks. The open-source release and practical guidance have substantial implications for accessible LLM development among smaller research teams and non-English language communities.

Abstract

Steel-LLM is a Chinese-centric language model developed from scratch with the goal of creating a high-quality, open-source model despite limited computational resources. Launched in March 2024, the project aimed to train a 1-billion-parameter model on a large-scale dataset, prioritizing transparency and the sharing of practical insights to assist others in the community. The training process primarily focused on Chinese data, with a small proportion of English data included, addressing gaps in existing open-source LLMs by providing a more detailed and practical account of the model-building journey. Steel-LLM has demonstrated competitive performance on benchmarks such as CEVAL and CMMLU, outperforming early models from larger institutions. This paper provides a comprehensive summary of the project's key contributions, including data collection, model design, training methodologies, and the challenges encountered along the way, offering a valuable resource for researchers and practitioners looking to develop their own LLMs. The model checkpoints and training script are available at https://github.com/zhanshijinwat/Steel-LLM.

Paper Structure

This paper contains 25 sections, 2 equations, 5 figures, 8 tables.

Table of Contents

  1. Introduction
  2. Related Works
  3. Architecture
  4. Soft MoE
  5. Enhanced Feed-Forward Network
  6. Training Framework
  7. Pretraining
  8. Post training
  9. Supervised Finetuning
  10. Ablation Studies and Evaluation
  11. Experiment 1: Full Infinity-Instruct + Wanjuan MCQ
  12. Experiment 2: 700K Chinese Infinity-Instruct + Wanjuan MCQ
  13. Experiment 3: Chinese + 100% English Data
  14. Experiment 4: Chinese + 20% English Data (Balanced)
  15. Experiment 5: Chinese + 20% English + English MCQ
  16. ...and 10 more sections

Figures (5)

  • Figure 1: The architecture of Steel-LLM Transformer block
  • Figure 2: Pretraining Data Distribution
  • Figure 3: Pre-training loss curve for Steel-LLM
  • Figure 4: Supervised Fine-tuning loss curve for Steel-LLM
  • Figure 5: Direct Preference Optimization loss curve for Steel-LLM