Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

DNA 1.0 Technical Report

Jungyup Lee, Jemin Kim, Sang Park, SeungJae Lee

TL;DR

DNA 1.0 8B Instruct presents a compact bilingual LLM built on Llama 3.1 8B, optimized for Korean and English through continual pre-training, supervised fine-tuning, and targeted post-training. The pipeline combines SLERP-based model merging, direct preference optimization, and knowledge distillation to transfer knowledge from larger teachers, achieving state-of-the-art results on Korean benchmarks while maintaining strong English performance. Extensive evaluation across four benchmark groups demonstrates strong Korean specialization (KMMLU, KoBEST) and competitive multilingual and math/science capabilities (MMLU, GSM8K, GPQA), alongside robust long-context processing up to 32K tokens. The model is open-source and designed for efficient deployment, underscoring DNA’s potential as a practical, language-specific yet broadly capable bilingual LLM.

Abstract

In this report, we present DNA 1.0 8B Instruct, a state-of-the-art bilingual language model optimized for Korean and English language tasks. By applying continual pre-training (CPT) with high-quality Korean datasets to Llama 3.1 8B and subsequent supervised fine-tuning (SFT), we create an instruction-following model with enhanced Korean language capabilities. This model is then merged with Llama 3.1 8B Instruct via spherical linear interpolation (SLERP) and undergoes further optimization through direct preference optimization (DPO) and knowledge distillation (KD). DNA 1.0 8B Instruct achieves state-of-the-art results on Korean-specific tasks, including KMMLU (53.26%), KoBEST (83.40%), and BELEBELE (57.99%), while maintaining strong English capabilities on MMLU (66.64%), MMLU-Pro (43.05%) and GSM8K (80.52%). As an open model, DNA 1.0 8B Instruct represents a significant advancement in bilingual language modeling. As an open model, DNA 1.0 8B Instruct is freely available through https://huggingface.co/dnotitia/Llama-DNA-1.0-8B-Instruct . For commercial licensing inquiries or feedback, please contact us at https://www.dnotitia.com/contact/post-form

DNA 1.0 Technical Report

TL;DR

DNA 1.0 8B Instruct presents a compact bilingual LLM built on Llama 3.1 8B, optimized for Korean and English through continual pre-training, supervised fine-tuning, and targeted post-training. The pipeline combines SLERP-based model merging, direct preference optimization, and knowledge distillation to transfer knowledge from larger teachers, achieving state-of-the-art results on Korean benchmarks while maintaining strong English performance. Extensive evaluation across four benchmark groups demonstrates strong Korean specialization (KMMLU, KoBEST) and competitive multilingual and math/science capabilities (MMLU, GSM8K, GPQA), alongside robust long-context processing up to 32K tokens. The model is open-source and designed for efficient deployment, underscoring DNA’s potential as a practical, language-specific yet broadly capable bilingual LLM.

Abstract

In this report, we present DNA 1.0 8B Instruct, a state-of-the-art bilingual language model optimized for Korean and English language tasks. By applying continual pre-training (CPT) with high-quality Korean datasets to Llama 3.1 8B and subsequent supervised fine-tuning (SFT), we create an instruction-following model with enhanced Korean language capabilities. This model is then merged with Llama 3.1 8B Instruct via spherical linear interpolation (SLERP) and undergoes further optimization through direct preference optimization (DPO) and knowledge distillation (KD). DNA 1.0 8B Instruct achieves state-of-the-art results on Korean-specific tasks, including KMMLU (53.26%), KoBEST (83.40%), and BELEBELE (57.99%), while maintaining strong English capabilities on MMLU (66.64%), MMLU-Pro (43.05%) and GSM8K (80.52%). As an open model, DNA 1.0 8B Instruct represents a significant advancement in bilingual language modeling. As an open model, DNA 1.0 8B Instruct is freely available through https://huggingface.co/dnotitia/Llama-DNA-1.0-8B-Instruct . For commercial licensing inquiries or feedback, please contact us at https://www.dnotitia.com/contact/post-form
Paper Structure (23 sections, 2 figures, 8 tables)

This paper contains 23 sections, 2 figures, 8 tables.

Table of Contents

  1. Introduction
  2. Model Architecure
  3. Pre-training
  4. Pre-training Data
  5. Continual Pre-training
  6. Post-training
  7. Post-training data
  8. Supervised Fine-tuning Data
  9. Preference Data
  10. Supervised Fine-tuning (SFT)
  11. Data Preparation and Curation
  12. Training Configuration
  13. Spherical Linear Interpolation (SLERP)
  14. Direct Preference Optimization (DPO)
  15. Knowledge Distillation (KD)
  16. ...and 8 more sections

Figures (2)

  • Figure 1: Overview of DNA 1.0 8B Instruct training pipeline. The process includes continual pre-training (CPT), supervised fine-tuning (SFT), model merging via SLERP, and final optimization through direct preference optimization (DPO) and knowledge distillation (KD).
  • Figure 2: Evaluation results on the "Needle In A Haystack" (NIAH) tests. The x-axis represents the token length of the input text, while the y-axis shows the relative position within the text. DNA 1.0 8B Instruct performs well across all context window length up to 32K.