Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Stratos: An End-to-End Distillation Pipeline for Customized LLMs under Distributed Cloud Environments

Ziming Dai, Tuo Zhang, Fei Gao, Xingyi Cai, Xiaofei Wang, Cheng Zhang, Wenyu Wang, Chengjie Zang

TL;DR

Stratos tackles the practical problem of creating customized, cost-efficient LLMs in distributed clouds under constraints such as latency and budget. It proposes an end-to-end pipeline that automates server selection, teacher–student pairing, and distillation strategy using a multi-objective optimization framework with Pareto-front guidance. In experiments on GSM8K, AIME, and a Mahjong domain task, distilled students achieve up to 4× higher accuracy than the teacher baseline while reducing latency and cost, including successful knowledge injection beyond teacher knowledge. The approach offers a practical toolkit for vertical-domain LLM deployment at scale.

Abstract

The growing industrial demand for customized and cost-efficient large language models (LLMs) is fueled by the rise of vertical, domain-specific tasks and the need to optimize performance under constraints such as latency and budget. Knowledge distillation, as an efficient model compression and transfer technique, offers a feasible solution. However, existing distillation frameworks often require manual intervention and struggle to meet such complex user-defined distillation requirements. To bridge this gap, we propose Stratos, an end-to-end LLM distillation pipeline that automates server and model selection, knowledge distillation, and deployment in distributed cloud environments. Given user-defined constraints on model performance and system budget, Stratos automatically selects Pareto-optimal servers, dynamically matches teacher-student pairs, and adapts distillation strategies based on task complexity to optimize cloud hosting. Experiments show that Stratos produces a student model that achieves four times the accuracy of its GPT-4o teacher baseline on a rare, domain-specific Mahjong reasoning task with reverse synthetic data and knowledge injection. Moreover, it achieves reduced latency and cost without compromising accuracy. These results highlight its promise for vertical-domain LLM deployment.

Stratos: An End-to-End Distillation Pipeline for Customized LLMs under Distributed Cloud Environments

TL;DR

Stratos tackles the practical problem of creating customized, cost-efficient LLMs in distributed clouds under constraints such as latency and budget. It proposes an end-to-end pipeline that automates server selection, teacher–student pairing, and distillation strategy using a multi-objective optimization framework with Pareto-front guidance. In experiments on GSM8K, AIME, and a Mahjong domain task, distilled students achieve up to 4× higher accuracy than the teacher baseline while reducing latency and cost, including successful knowledge injection beyond teacher knowledge. The approach offers a practical toolkit for vertical-domain LLM deployment at scale.

Abstract

The growing industrial demand for customized and cost-efficient large language models (LLMs) is fueled by the rise of vertical, domain-specific tasks and the need to optimize performance under constraints such as latency and budget. Knowledge distillation, as an efficient model compression and transfer technique, offers a feasible solution. However, existing distillation frameworks often require manual intervention and struggle to meet such complex user-defined distillation requirements. To bridge this gap, we propose Stratos, an end-to-end LLM distillation pipeline that automates server and model selection, knowledge distillation, and deployment in distributed cloud environments. Given user-defined constraints on model performance and system budget, Stratos automatically selects Pareto-optimal servers, dynamically matches teacher-student pairs, and adapts distillation strategies based on task complexity to optimize cloud hosting. Experiments show that Stratos produces a student model that achieves four times the accuracy of its GPT-4o teacher baseline on a rare, domain-specific Mahjong reasoning task with reverse synthetic data and knowledge injection. Moreover, it achieves reduced latency and cost without compromising accuracy. These results highlight its promise for vertical-domain LLM deployment.
Paper Structure (15 sections, 2 equations, 5 figures, 5 tables)

This paper contains 15 sections, 2 equations, 5 figures, 5 tables.

Table of Contents

  1. Introduction
  2. Related Works
  3. Design Details
  4. Overview of Stratos
  5. Server Selector
  6. Teacher Selector
  7. Student Selector
  8. Knowledge Alignment
  9. Knowledge Injection
  10. Empirical Study
  11. Experimental Setup
  12. Strategy Effectiveness across Multi-Objective Metrics
  13. Distillation Performance within Teacher's Knowledge Domain
  14. Distillation Performance outside Teacher's Knowledge Domain
  15. Conclusion

Figures (5)

  • Figure 1: Industrial deployment distribution of LLMs by parameter size.
  • Figure 2: Real-world scenario and optimization entry point.
  • Figure 3: System overview of the proposed end-to-end distillation pipeline.
  • Figure 4: Illustration of Mahjong winning rules.
  • Figure 5: Example of synthetic Mahjong data generation.