Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Solar Open Technical Report

Sungrae Park, Sanghoon Kim, Jungho Cho, Gyoungjin Gim, Dawoon Jung, Mikyoung Cha, Eunhae Choo, Taekgyu Hong, Minbyul Jeong, SeHwan Joo, Minsoo Khang, Eunwon Kim, Minjeong Kim, Sujeong Kim, Yunsu Kim, Hyeonju Lee, Seunghyun Lee, Sukyung Lee, Siyoung Park, Gyungin Shin, Inseo Song, Wonho Song, Seonghoon Yang, Seungyoun Yi, Sanghoon Yoon, Jeonghyun Ko, Seyoung Song, Keunwoo Choi, Hwalsuk Lee, Sunghun Kim, Du-Seong Chang, Kyunghyun Cho, Junsuk Choe, Hwaran Lee, Jae-Gil Lee, KyungTae Lim, Alice Oh

TL;DR

Solar Open addresses data scarcity and reasoning for underserved languages by combining aggressive synthetic data generation, a bilingual curriculum, and a decoupled RL framework called SnapPO. The model employs a 102.6B MoE architecture with a Korean-optimized tokenizer and pre-trained on 20T tokens, including 4.5T synthetic data, then refined via mid-training RL and SFT. Evaluations across English and Korean benchmarks show competitive performance, with notable gains in Korean finance, law, and medical domains, and strong preference alignment. The approach provides a scalable blueprint for developing frontier capabilities in low-resource languages while maintaining general performance and safety.

Abstract

We introduce Solar Open, a 102B-parameter bilingual Mixture-of-Experts language model for underserved languages. Solar Open demonstrates a systematic methodology for building competitive LLMs by addressing three interconnected challenges. First, to train effectively despite data scarcity for underserved languages, we synthesize 4.5T tokens of high-quality, domain-specific, and RL-oriented data. Second, we coordinate this data through a progressive curriculum jointly optimizing composition, quality thresholds, and domain coverage across 20 trillion tokens. Third, to enable reasoning capabilities through scalable RL, we apply our proposed framework SnapPO for efficient optimization. Across benchmarks in English and Korean, Solar Open achieves competitive performance, demonstrating the effectiveness of this methodology for underserved language AI development.

Solar Open Technical Report

TL;DR

Solar Open addresses data scarcity and reasoning for underserved languages by combining aggressive synthetic data generation, a bilingual curriculum, and a decoupled RL framework called SnapPO. The model employs a 102.6B MoE architecture with a Korean-optimized tokenizer and pre-trained on 20T tokens, including 4.5T synthetic data, then refined via mid-training RL and SFT. Evaluations across English and Korean benchmarks show competitive performance, with notable gains in Korean finance, law, and medical domains, and strong preference alignment. The approach provides a scalable blueprint for developing frontier capabilities in low-resource languages while maintaining general performance and safety.

Abstract

We introduce Solar Open, a 102B-parameter bilingual Mixture-of-Experts language model for underserved languages. Solar Open demonstrates a systematic methodology for building competitive LLMs by addressing three interconnected challenges. First, to train effectively despite data scarcity for underserved languages, we synthesize 4.5T tokens of high-quality, domain-specific, and RL-oriented data. Second, we coordinate this data through a progressive curriculum jointly optimizing composition, quality thresholds, and domain coverage across 20 trillion tokens. Third, to enable reasoning capabilities through scalable RL, we apply our proposed framework SnapPO for efficient optimization. Across benchmarks in English and Korean, Solar Open achieves competitive performance, demonstrating the effectiveness of this methodology for underserved language AI development.
Paper Structure (68 sections, 5 figures, 7 tables)

This paper contains 68 sections, 5 figures, 7 tables.

Table of Contents

  1. Introduction
  2. Motivation
  3. Challenges and Our Solutions
  4. A. Synthetic Data for K-Data Scarcity and RL
  5. B. Bilingual and Reasoning-Targeted Curriculum
  6. C. Scalable RL Framework
  7. Interconnections
  8. Model Architecture
  9. Solar Open Tokenizer
  10. Algorithm and Normalization
  11. Chat Template
  12. Evaluation on Tokenizer Efficiency
  13. Mixture-of-Experts Architecture
  14. Architectural Design Process
  15. Distinctive Choices
  16. ...and 53 more sections

Figures (5)

  • Figure 1: Overall performance of Solar Open and other comparable models.
  • Figure 2: The compression rates of the Solar Open Tokenizer and other tokenizers (higher is more efficient). In each bar group, the nine bars left to Solar Open represent the 'global' (English and/or Chinese-centric) models, and the four bars right to Solar Open represent the Korean-centric models.
  • Figure 3: Inference-time tokenizer efficiency across languages and reasoning settings.
  • Figure 4: The data curriculum for the pre-training phases of Solar Open.
  • Figure 5: Training trajectory comparison between Solar Open and GLM-4.5-Base (23T tokens). Solar Open achieves comparable performance at 10.9T tokens (English) and 17.8T tokens (Korean), based on MMLU, MMLU-Pro, and HellaSwag benchmarks. Curves are smoothed for clarity.