Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

A.X K1 Technical Report

Sung Jun Cheon, Jaekyung Cho, Seongho Choi, Hyunjun Eun, Seokhwan Jo, Jaehyun Jun, Minsoo Kang, Jin Kim, Jiwon Kim, Minsang Kim, Sungwan Kim, Seungsik Kim, Tae Yoon Kim, Youngrang Kim, Hyeongmun Lee, Sangyeol Lee, Sungeun Lee, Youngsoon Lee, Yujin Lee, Seongmin Ok, Chanyong Park, Hyewoong Park, Junyoung Park, Hyunho Yang, Subin Yi, Soohyun Bae, Dhammiko Arya, Yongseok Choi, Sangho Choi, Dongyeon Cho, Seungmo Cho, Gyoungeun Han, Yong-jin Han, Seokyoung Hong, Hyeon Hwang, Wonbeom Jang, Minjeong Ju, Wonjin Jung, Keummin Ka, Sungil Kang, Dongnam Kim, Joonghoon Kim, Jonghwi Kim, SaeRom Kim, Sangjin Kim, Seongwon Kim, Youngjin Kim, Seojin Lee, Sunwoo Lee, Taehoon Lee, Chanwoo Park, Sohee Park, Sooyeon Park, Yohan Ra, Sereimony Sek, Seungyeon Seo, Gun Song, Sanghoon Woo, Janghan Yoon, Sungbin Yoon

TL;DR

A.X K1 introduces a compute-efficient $519$B Mixture-of-Experts model with $33$B active parameters and a $160K$ vocabulary, trained on ~10T tokens under a fixed FLOP budget. The authors design Think-Fusion to enable explicit switching between thinking and concise responses within a single checkpoint, integrating linear model merging, mode-overlap data, and GSPO-based on-policy reinforcement learning. The architecture uses Multi-head Latent Attention (MLA) with careful normalization, long-context curriculum up to $32K$, and system-level optimizations across 1,536 H200 GPUs to achieve competitive performance on English and Korean benchmarks, especially in math and Korean language tasks. The work demonstrates practical scalability for frontier-scale MoE systems under real-world constraints and outlines future directions toward multimodal capabilities and larger-scale deployment.

Abstract

We introduce A.X K1, a 519B-parameter Mixture-of-Experts (MoE) language model trained from scratch. Our design leverages scaling laws to optimize training configurations and vocabulary size under fixed computational budgets. A.X K1 is pre-trained on a corpus of approximately 10T tokens, curated by a multi-stage data processing pipeline. Designed to bridge the gap between reasoning capability and inference efficiency, A.X K1 supports explicitly controllable reasoning to facilitate scalable deployment across diverse real-world scenarios. We propose a simple yet effective Think-Fusion training recipe, enabling user-controlled switching between thinking and non-thinking modes within a single unified model. Extensive evaluations demonstrate that A.X K1 achieves performance competitive with leading open-source models, while establishing a distinctive advantage in Korean-language benchmarks.

A.X K1 Technical Report

TL;DR

A.X K1 introduces a compute-efficient B Mixture-of-Experts model with B active parameters and a vocabulary, trained on ~10T tokens under a fixed FLOP budget. The authors design Think-Fusion to enable explicit switching between thinking and concise responses within a single checkpoint, integrating linear model merging, mode-overlap data, and GSPO-based on-policy reinforcement learning. The architecture uses Multi-head Latent Attention (MLA) with careful normalization, long-context curriculum up to , and system-level optimizations across 1,536 H200 GPUs to achieve competitive performance on English and Korean benchmarks, especially in math and Korean language tasks. The work demonstrates practical scalability for frontier-scale MoE systems under real-world constraints and outlines future directions toward multimodal capabilities and larger-scale deployment.

Abstract

We introduce A.X K1, a 519B-parameter Mixture-of-Experts (MoE) language model trained from scratch. Our design leverages scaling laws to optimize training configurations and vocabulary size under fixed computational budgets. A.X K1 is pre-trained on a corpus of approximately 10T tokens, curated by a multi-stage data processing pipeline. Designed to bridge the gap between reasoning capability and inference efficiency, A.X K1 supports explicitly controllable reasoning to facilitate scalable deployment across diverse real-world scenarios. We propose a simple yet effective Think-Fusion training recipe, enabling user-controlled switching between thinking and non-thinking modes within a single unified model. Extensive evaluations demonstrate that A.X K1 achieves performance competitive with leading open-source models, while establishing a distinctive advantage in Korean-language benchmarks.
Paper Structure (43 sections, 3 equations, 6 figures, 7 tables)

This paper contains 43 sections, 3 equations, 6 figures, 7 tables.

Table of Contents

  1. Introduction
  2. Organization.
  3. Architecture
  4. Resource-Constrained Model Design
  5. Model Configuration
  6. Tokenizer
  7. Design of Vocabulary Size.
  8. Pre-training
  9. Pre-training Dataset
  10. Document parsing pipeline.
  11. Synthetic data pipeline.
  12. Data curation pipeline.
  13. Pre-training Process
  14. Hyperparameters
  15. Parallelism Configuration
  16. ...and 28 more sections

Figures (6)

  • Figure 1: Training loss curves for Pre-Normalization and Dual-Normalization during early training of A.X K1 Light (20B-A3B, small-scale model for dry-run).
  • Figure 2: A Data Processing Framework for Constructing High-Quality Pre-training Datasets. The diagram illustrates the transformation of raw data sources (documents and raw data) into a final training corpus through document parsing, synthetic data generation, and a multi-stage curation pipeline (including cleaning, anonymization, deduplication, and diverse classifiers).
  • Figure 3: One example of evaluation prompt for AIME25. The model’s response is evaluated by comparing the ANSWER field against the ground-truth answer.
  • Figure 4: One example of evaluation prompt for HRM8K. The model’s response is evaluated by comparing the ANSWER field against the ground-truth answer.
  • Figure 5: One example of an evaluation prompt for MMLU. The model's response is evaluated by comparing the [the_answer_letter] field against the ground-truth answer.
  • ...and 1 more figures