Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Chinese Tiny LLM: Pretraining a Chinese-Centric Large Language Model

Xinrun Du, Zhouliang Yu, Songyang Gao, Ding Pan, Yuyang Cheng, Ziyang Ma, Ruibin Yuan, Xingwei Qu, Jiaheng Liu, Tianyu Zheng, Xinchen Luo, Guorui Zhou, Wenhu Chen, Ge Zhang

TL;DR

CT-LLM introduces a 2B Chinese-centric LLM pretrained on a large, Chinese-dominated corpus (MAP-CC) to elevate Chinese language capabilities from the ground up. It combines extensive pretraining data with SFT and preference-based alignment (DPO), plus a new Chinese hard-case benchmark (CHC-Bench) to evaluate instruction following in Chinese. The work reports competitive cross-lingual performance, strong Chinese proficiency, safety benefits, and cultural-bias insights for a model trained predominantly on Chinese data, while openly releasing its data processing pipeline and benchmarks. This approach challenges English-centric paradigms and offers a scalable path toward more inclusive, language-diverse large language models with practical open-source value.

Abstract

In this study, we introduce CT-LLM, a 2B large language model (LLM) that illustrates a pivotal shift towards prioritizing the Chinese language in developing LLMs. Uniquely initiated from scratch, CT-LLM diverges from the conventional methodology by primarily incorporating Chinese textual data, utilizing an extensive corpus of 1,200 billion tokens, including 800 billion Chinese tokens, 300 billion English tokens, and 100 billion code tokens. This strategic composition facilitates the model's exceptional proficiency in understanding and processing Chinese, a capability further enhanced through alignment techniques. Demonstrating remarkable performance on the CHC-Bench, CT-LLM excels in Chinese language tasks, and showcases its adeptness in English through SFT. This research challenges the prevailing paradigm of training LLMs predominantly on English corpora and then adapting them to other languages, broadening the horizons for LLM training methodologies. By open-sourcing the full process of training a Chinese LLM, including a detailed data processing procedure with the obtained Massive Appropriate Pretraining Chinese Corpus (MAP-CC), a well-chosen multidisciplinary Chinese Hard Case Benchmark (CHC-Bench), and the 2B-size Chinese Tiny LLM (CT-LLM), we aim to foster further exploration and innovation in both academia and industry, paving the way for more inclusive and versatile language models.

Chinese Tiny LLM: Pretraining a Chinese-Centric Large Language Model

TL;DR

CT-LLM introduces a 2B Chinese-centric LLM pretrained on a large, Chinese-dominated corpus (MAP-CC) to elevate Chinese language capabilities from the ground up. It combines extensive pretraining data with SFT and preference-based alignment (DPO), plus a new Chinese hard-case benchmark (CHC-Bench) to evaluate instruction following in Chinese. The work reports competitive cross-lingual performance, strong Chinese proficiency, safety benefits, and cultural-bias insights for a model trained predominantly on Chinese data, while openly releasing its data processing pipeline and benchmarks. This approach challenges English-centric paradigms and offers a scalable path toward more inclusive, language-diverse large language models with practical open-source value.

Abstract

In this study, we introduce CT-LLM, a 2B large language model (LLM) that illustrates a pivotal shift towards prioritizing the Chinese language in developing LLMs. Uniquely initiated from scratch, CT-LLM diverges from the conventional methodology by primarily incorporating Chinese textual data, utilizing an extensive corpus of 1,200 billion tokens, including 800 billion Chinese tokens, 300 billion English tokens, and 100 billion code tokens. This strategic composition facilitates the model's exceptional proficiency in understanding and processing Chinese, a capability further enhanced through alignment techniques. Demonstrating remarkable performance on the CHC-Bench, CT-LLM excels in Chinese language tasks, and showcases its adeptness in English through SFT. This research challenges the prevailing paradigm of training LLMs predominantly on English corpora and then adapting them to other languages, broadening the horizons for LLM training methodologies. By open-sourcing the full process of training a Chinese LLM, including a detailed data processing procedure with the obtained Massive Appropriate Pretraining Chinese Corpus (MAP-CC), a well-chosen multidisciplinary Chinese Hard Case Benchmark (CHC-Bench), and the 2B-size Chinese Tiny LLM (CT-LLM), we aim to foster further exploration and innovation in both academia and industry, paving the way for more inclusive and versatile language models.
Paper Structure (30 sections, 7 figures, 13 tables)

This paper contains 30 sections, 7 figures, 13 tables.

Table of Contents

  1. Introduction
  2. Related Works
  3. LLM with Chinese Language Ability
  4. Chinese Corpora for Pretraining and Alignment
  5. Emergence of Multilingual Capacity
  6. Pretraining
  7. Data
  8. Model Architecture
  9. Supervised Finetuning
  10. Learning from Human Preferences
  11. Evaluations
  12. Results of Metrics
  13. Evaluation Datasets and Metrics
  14. Training Process and Comparative Analysis
  15. Safety Evaluation
  16. ...and 15 more sections

Figures (7)

  • Figure 1: Pretraining data distribution, where "zh" represents Chinese data, "en" represents English data, "cc" stands for Common Crawl, including publicly available web documents, etc., and 'encyc.' refers to the encyclopedia.
  • Figure 2: Above is the data processing flow and deduplication ratios, below is a schematic diagram of similar line deduplication.
  • Figure 3: Political spectrum positioning of CT-LLM compared to other open-source models. We test the models' orientation with the benchmark introduced by feng2023pretraining.
  • Figure 4: Average Reward for Rejected Responses
  • Figure 5: Average Reward for Chosen Responses
  • ...and 2 more figures