Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Ruyi2 Technical Report

Huan Song, Shuyu Tian, Junyi Hao, Minxiu Xu, Hongjun An, Yiliang Song, Jiawei Shao, Xuelong Li

TL;DR

Ruyi2 is introduced as an evolution of the AI Flow framework, and results confirm that family-based parameter sharing is a highly effective strategy, establishing a new "Train Once, Deploy Many" paradigms and providing a key reference for balancing architectural efficiency with high-performance capabilities.

Abstract

Large Language Models (LLMs) face significant challenges regarding deployment costs and latency, necessitating adaptive computing strategies. Building upon the AI Flow framework, we introduce Ruyi2 as an evolution of our adaptive model series designed for efficient variable-depth computation. While early-exit architectures offer a viable efficiency-performance balance, the Ruyi model and existing methods often struggle with optimization complexity and compatibility with large-scale distributed training. To bridge this gap, Ruyi2 introduces a stable "Familial Model" based on Megatron-LM. By using 3D parallel training, it achieves a 2-3 times speedup over Ruyi, while performing comparably to same-sized Qwen3 models. These results confirm that family-based parameter sharing is a highly effective strategy, establishing a new "Train Once, Deploy Many" paradigm and providing a key reference for balancing architectural efficiency with high-performance capabilities.

Ruyi2 Technical Report

TL;DR

Ruyi2 is introduced as an evolution of the AI Flow framework, and results confirm that family-based parameter sharing is a highly effective strategy, establishing a new "Train Once, Deploy Many" paradigms and providing a key reference for balancing architectural efficiency with high-performance capabilities.

Abstract

Large Language Models (LLMs) face significant challenges regarding deployment costs and latency, necessitating adaptive computing strategies. Building upon the AI Flow framework, we introduce Ruyi2 as an evolution of our adaptive model series designed for efficient variable-depth computation. While early-exit architectures offer a viable efficiency-performance balance, the Ruyi model and existing methods often struggle with optimization complexity and compatibility with large-scale distributed training. To bridge this gap, Ruyi2 introduces a stable "Familial Model" based on Megatron-LM. By using 3D parallel training, it achieves a 2-3 times speedup over Ruyi, while performing comparably to same-sized Qwen3 models. These results confirm that family-based parameter sharing is a highly effective strategy, establishing a new "Train Once, Deploy Many" paradigm and providing a key reference for balancing architectural efficiency with high-performance capabilities.
Paper Structure (26 sections, 7 equations, 6 figures, 2 tables)

This paper contains 26 sections, 7 equations, 6 figures, 2 tables.

Table of Contents

  1. Introduction
  2. Architecture
  3. Pre-training
  4. Data Acquisition
  5. Data Processing
  6. Pre-training Strategy
  7. Multi-stage Continual Pre-Training Strategy
  8. Joint Optimization Framework
  9. Post-training
  10. Overall Post-training Pipeline
  11. Post-training Data
  12. Post-training Strategy
  13. Multi-stage Instruction Alignment
  14. 1.7B Architecture Specific Optimization
  15. Architecture Innovation: The DaE Framework
  16. ...and 11 more sections

Figures (6)

  • Figure 1: The data engineering pipeline for constructing a high-quality training-ready corpus. The pipeline consists of four main stages: filtering, deduplication, decontamination, and data mixture, ensuring the high density and quality of the dataset.
  • Figure 2: Overview of the Staged Capability Improvement Pipeline
  • Figure 3: SFT Data Mixture Strategy: Evolution from Diverse Generalization to Focused Domain Specialization
  • Figure 4: Two stages of DaE
  • Figure 5: Input/output cosine similarity score of each token per layer for text "A fox sat on a box".
  • ...and 1 more figures