Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Mastering Text, Code and Math Simultaneously via Fusing Highly Specialized Language Models

Ning Ding, Yulin Chen, Ganqu Cui, Xingtai Lv, Weilin Zhao, Ruobing Xie, Bowen Zhou, Zhiyuan Liu, Maosong Sun

TL;DR

UltraFuser addresses the challenge of achieving simultaneous proficiency across text, code, and math by fusing three domain-specialized LLMs through a token-level gating mechanism. It introduces a two-stage training procedure with balanced data sampling and a high-quality instruction-tuning dataset, UltraChat 2, to align the fused model with multi-domain tasks. Empirical results across language, code, and math benchmarks show that the fused model can outperform individual specialists and naive joint training, demonstrating the viability and benefits of explicit specialist fusion for broad-domain expertise. The work provides a practical framework and resources for integrating specialized capabilities into general-purpose interactive systems, with implications for multi-domain AI tooling and future extensions to additional symbol systems.

Abstract

Underlying data distributions of natural language, programming code, and mathematical symbols vary vastly, presenting a complex challenge for large language models (LLMs) that strive to achieve high performance across all three domains simultaneously. Achieving a very high level of proficiency for an LLM within a specific domain often requires extensive training with relevant corpora, which is typically accompanied by a sacrifice in performance in other domains. In this paper, we propose to fuse models that are already highly-specialized directly. The proposed fusing framework, UltraFuser, consists of three distinct specialists that are already sufficiently trained on language, coding, and mathematics. A token-level gating mechanism is introduced to blend the specialists' outputs. A two-stage training strategy accompanied by balanced sampling is designed to ensure stability. To effectively train the fused model, we further construct a high-quality supervised instruction tuning dataset, UltraChat 2, which includes text, code, and mathematical content. This dataset comprises approximately 300,000 instructions and covers a wide range of topics in each domain. Experiments show that our model could simultaneously achieve mastery of the three crucial domains.

Mastering Text, Code and Math Simultaneously via Fusing Highly Specialized Language Models

TL;DR

UltraFuser addresses the challenge of achieving simultaneous proficiency across text, code, and math by fusing three domain-specialized LLMs through a token-level gating mechanism. It introduces a two-stage training procedure with balanced data sampling and a high-quality instruction-tuning dataset, UltraChat 2, to align the fused model with multi-domain tasks. Empirical results across language, code, and math benchmarks show that the fused model can outperform individual specialists and naive joint training, demonstrating the viability and benefits of explicit specialist fusion for broad-domain expertise. The work provides a practical framework and resources for integrating specialized capabilities into general-purpose interactive systems, with implications for multi-domain AI tooling and future extensions to additional symbol systems.

Abstract

Underlying data distributions of natural language, programming code, and mathematical symbols vary vastly, presenting a complex challenge for large language models (LLMs) that strive to achieve high performance across all three domains simultaneously. Achieving a very high level of proficiency for an LLM within a specific domain often requires extensive training with relevant corpora, which is typically accompanied by a sacrifice in performance in other domains. In this paper, we propose to fuse models that are already highly-specialized directly. The proposed fusing framework, UltraFuser, consists of three distinct specialists that are already sufficiently trained on language, coding, and mathematics. A token-level gating mechanism is introduced to blend the specialists' outputs. A two-stage training strategy accompanied by balanced sampling is designed to ensure stability. To effectively train the fused model, we further construct a high-quality supervised instruction tuning dataset, UltraChat 2, which includes text, code, and mathematical content. This dataset comprises approximately 300,000 instructions and covers a wide range of topics in each domain. Experiments show that our model could simultaneously achieve mastery of the three crucial domains.
Paper Structure (16 sections, 2 equations, 8 figures, 8 tables, 1 algorithm)

This paper contains 16 sections, 2 equations, 8 figures, 8 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Related Work
  3. Our Approach
  4. Model
  5. UltraChat 2 Dataset
  6. Experiments
  7. Experimental Settings
  8. Results on Benchmarks
  9. Results on MT-Bench
  10. Ablation Study
  11. Expertise Analysis
  12. Conclusion
  13. Implementation Details
  14. Efficient Inference
  15. Disscussion and Limitations
  16. ...and 1 more sections

Figures (8)

  • Figure 1: Performance on three different domains of specialized models and our UltraFuser. The performance for the text domain is computed by the average results on TruthfulQA (Acc) lin2021truthfulqa and AlpacaEval (Win Rate) alpaca_eval datasets; the performance for the code domain is Pass@1 of HumanEval chen2021evaluating; and the performance for the math domain is the average result of GSM8K (Pass@1) cobbe2021training, MATH (Pass@1) hendrycks2021measuring, SAT-Math (Acc) zhong2023agieval, and AQuA-RAT (Acc) ling-etal-2017-program datasets. All numbers are zero-shot results.
  • Figure 2: Architectrue of our proposed UltraFuser framework. We do not show the two-stage training in this illustration.
  • Figure 3: t-SNE visualization of UltraChat 2 dataset.
  • Figure 4: Performance comparison of Llama-2 model trained on UltraChat and UltraChat 2. The performance for the text domain is computed by the average results on TruthfulQA (Acc) and AlpacaEval (Win Rate) datasets; the performance for the code domain is Pass@1 of HumanEval; and the performance for the math domain is the average result of GSM8K (Pass@1) and MATH (Pass@1).
  • Figure 5: Performance comparisons between specialist models and the further training versions of them.
  • ...and 3 more figures