Panda LLM: Training Data and Evaluation for Open-Sourced Chinese Instruction-Following Large Language Models

TL;DR

Open-source Chinese instruction-following LLMs face trust, transparency, and data-recipe challenges. The paper presents Panda LLM, built on LLaMA bases, with a two-stage training pipeline combining Chinese corpora with COIG for instruction-tuning, and releases weight deltas due to licensing. Key contributions include a first comparative evaluation of open-source Chinese LLMs, demonstration that COIG-driven instruction tuning yields large gains, and public release of data, checkpoints, and code. The work has practical impact by enabling broader access to high-quality, trustworthy Chinese LLMs and providing a blueprint for data strategies in instruction-following models.

Abstract

This project focuses on enhancing open-source large language models through instruction-tuning and providing comprehensive evaluations of their performance. We explore how various training data factors, such as quantity, quality, and linguistic distribution, influence the performance of instruction-tuned models trained on publicly accessible high-quality instruction datasets for both English and Chinese languages. Our goal is to supplement evaluation with quantitative analyses, providing valuable insights for the continued advancement of open-source chat models. Our model, data, and code are publicly available for others to use and build upon.

Figures (2)

• Figure 1: Illustrations of our proposed method.
• Figure 2: Train steps versus losses on (a). Training on NLP Chinese Corpus dataset, and (b). Training on COIG dataset.