Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Synthesizing Post-Training Data for LLMs through Multi-Agent Simulation

Shuo Tang, Xianghe Pang, Zexi Liu, Bohan Tang, Rui Ye, Tian Jin, Xiaowen Dong, Yanfeng Wang, Siheng Chen

TL;DR

The paper introduces MATRIX, a large-scale multi-agent simulator that grounds synthetic data in realistic, real-world-like scenarios, coupled with MATRIX-Gen, a scenario-driven instruction generator for post-training data. By simulating diverse agent interactions and leveraging scenario contexts, the framework produces high-quality SFT, DPO, reasoning, and domain-specific data. Empirical results show that models trained with as few as 20K MATRIX-Gen samples can outperform models trained on orders of magnitude more data on benchmarks like AlpacaEval 2 and Arena-Hard, underscoring the practical impact of grounded, scalable data synthesis. The work highlights scalability and realism as key levers for improving instruction-following capabilities while noting computational cost as a current limitation and avenue for future optimization.

Abstract

Post-training is essential for enabling large language models (LLMs) to follow human instructions. However, its effectiveness depends on high-quality instruction data, which is challenging to obtain in the real world due to privacy concerns, data scarcity, and high annotation costs. To fill this gap, inspired by the recent success of using LLMs to simulate human society, we propose MATRIX, a multi-agent simulator that automatically generates diverse text-based scenarios, capturing a wide range of real-world human needs in a realistic and scalable manner. Leveraging these outputs, we introduce a novel scenario-driven instruction generator MATRIX-Gen for controllable and highly realistic data synthesis. Extensive experiments demonstrate that our framework effectively generates both general and domain-specific data. On AlpacaEval 2 and Arena-Hard benchmarks, Llama-3-8B-Base, post-trained on datasets synthesized by MATRIX-Gen with just 20K instruction-response pairs, outperforms Meta's Llama-3-8B-Instruct model, which was trained on over 10M pairs.

Synthesizing Post-Training Data for LLMs through Multi-Agent Simulation

TL;DR

The paper introduces MATRIX, a large-scale multi-agent simulator that grounds synthetic data in realistic, real-world-like scenarios, coupled with MATRIX-Gen, a scenario-driven instruction generator for post-training data. By simulating diverse agent interactions and leveraging scenario contexts, the framework produces high-quality SFT, DPO, reasoning, and domain-specific data. Empirical results show that models trained with as few as 20K MATRIX-Gen samples can outperform models trained on orders of magnitude more data on benchmarks like AlpacaEval 2 and Arena-Hard, underscoring the practical impact of grounded, scalable data synthesis. The work highlights scalability and realism as key levers for improving instruction-following capabilities while noting computational cost as a current limitation and avenue for future optimization.

Abstract

Post-training is essential for enabling large language models (LLMs) to follow human instructions. However, its effectiveness depends on high-quality instruction data, which is challenging to obtain in the real world due to privacy concerns, data scarcity, and high annotation costs. To fill this gap, inspired by the recent success of using LLMs to simulate human society, we propose MATRIX, a multi-agent simulator that automatically generates diverse text-based scenarios, capturing a wide range of real-world human needs in a realistic and scalable manner. Leveraging these outputs, we introduce a novel scenario-driven instruction generator MATRIX-Gen for controllable and highly realistic data synthesis. Extensive experiments demonstrate that our framework effectively generates both general and domain-specific data. On AlpacaEval 2 and Arena-Hard benchmarks, Llama-3-8B-Base, post-trained on datasets synthesized by MATRIX-Gen with just 20K instruction-response pairs, outperforms Meta's Llama-3-8B-Instruct model, which was trained on over 10M pairs.

Paper Structure

This paper contains 28 sections, 1 equation, 12 figures, 22 tables.

Table of Contents

  1. Introduction
  2. Proposed Post-Training System
  3. MATRIX: Multi-Agent Simulator
  4. Real-World-Grounded Agents
  5. Homophily-Guided Communication
  6. Scenario Generation
  7. Discussions
  8. Experimental Results
  9. Experimental Setups
  10. Evaluation in General Domains
  11. Evaluation in Specific Domains
  12. Analysis
  13. Related Works
  14. Conclusions
  15. Limitations
  16. ...and 13 more sections

Figures (12)

  • Figure 1: Overview of the data synthesis system.
  • Figure 2: Tag cloud of agent profiles.
  • Figure 3: An example of persona and scenario that are used to synthesize an instruction.
  • Figure 4: Overview of the proposed post-training data generation process from scenarios.
  • Figure 5: Performance comparisons in specific domains, including code, multi-turn dialog, and safety.
  • ...and 7 more figures