Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Supercharging Packet-level Network Simulation of Large Model Training via Memoization and Fast-Forwarding

Fei Long, Kaihui Gao, Li Chen, Dan Li, Yiwei Zhang, Fei Gui, Yitao Xing, Wenjia Wei, Bingyang Liu

TL;DR

This paper tackles the slow runtime of packet-level DES for large-scale LLM training by exploiting two traffic regularities: repeated contention patterns and steady-states. It introduces Wormhole, a PLDES kernel that partitions networks by ports, memoizes unsteady-state transients via Flow Conflict Graphs, and fast-forwards steady-state periods using a rate-based identification scheme with provable error bounds. Key contributions include the port-level network partitioning algorithm, the Flow Conflict Graph-based unsteady-state memoization, the steady-state identification framework with theoretical error guarantees, and an ns-3 implementation that integrates with Unison for orthogonal parallelism. Empirical results show up to 1012× speedups with <1% end-to-end FCT error, and real-trace experiments confirm practical effectiveness, significantly reducing simulation time for GPT-13B-like workloads while preserving fidelity.

Abstract

Packet-level discrete-event simulation (PLDES) is a prevalent tool for evaluating detailed performance of large model training. Although PLDES offers high fidelity and generality, its slow performance has plagued networking practitioners. Existing optimization techniques either simplify the network model, resulting in large errors; or execute it in parallel using multiple processors, with an upper bound on speedup. This paper explores an alternative optimization direction that reduces the computational loads of PLDES while maintaining high fidelity. Our key insight is that, in distributed LLM training, packet-level traffic behaviors often exhibit repetitive contention patterns and steady-states where flow rates stabilize, ignoring these redundant discrete events speeds up the simulation considerably and the error is negligible. We realize this idea by proposing Wormhole, a user-transparent PLDES kernel capable of automatically memoization for unsteady-states and skipping for steady-states. Wormhole adopts network partitioning, state memoization and reuse, and rate-based steady-state identification to accurately determine the periods of each flow's steady-state, while maintaining simulation consistency after fast-forwarding. Experiments demonstrate that Wormhole can achieve a 744x speedup over the original ns-3 (510x for MoE workload), with a bounded error of <1%. Applying current multithreading parallel techniques and Wormhole together allows a 1012x speedup, reducing the simulation time for one GPT-13B training under 128 GPUs from 9 hours to 5 minutes.

Supercharging Packet-level Network Simulation of Large Model Training via Memoization and Fast-Forwarding

TL;DR

This paper tackles the slow runtime of packet-level DES for large-scale LLM training by exploiting two traffic regularities: repeated contention patterns and steady-states. It introduces Wormhole, a PLDES kernel that partitions networks by ports, memoizes unsteady-state transients via Flow Conflict Graphs, and fast-forwards steady-state periods using a rate-based identification scheme with provable error bounds. Key contributions include the port-level network partitioning algorithm, the Flow Conflict Graph-based unsteady-state memoization, the steady-state identification framework with theoretical error guarantees, and an ns-3 implementation that integrates with Unison for orthogonal parallelism. Empirical results show up to 1012× speedups with <1% end-to-end FCT error, and real-trace experiments confirm practical effectiveness, significantly reducing simulation time for GPT-13B-like workloads while preserving fidelity.

Abstract

Packet-level discrete-event simulation (PLDES) is a prevalent tool for evaluating detailed performance of large model training. Although PLDES offers high fidelity and generality, its slow performance has plagued networking practitioners. Existing optimization techniques either simplify the network model, resulting in large errors; or execute it in parallel using multiple processors, with an upper bound on speedup. This paper explores an alternative optimization direction that reduces the computational loads of PLDES while maintaining high fidelity. Our key insight is that, in distributed LLM training, packet-level traffic behaviors often exhibit repetitive contention patterns and steady-states where flow rates stabilize, ignoring these redundant discrete events speeds up the simulation considerably and the error is negligible. We realize this idea by proposing Wormhole, a user-transparent PLDES kernel capable of automatically memoization for unsteady-states and skipping for steady-states. Wormhole adopts network partitioning, state memoization and reuse, and rate-based steady-state identification to accurately determine the periods of each flow's steady-state, while maintaining simulation consistency after fast-forwarding. Experiments demonstrate that Wormhole can achieve a 744x speedup over the original ns-3 (510x for MoE workload), with a bounded error of <1%. Applying current multithreading parallel techniques and Wormhole together allows a 1012x speedup, reducing the simulation time for one GPT-13B training under 128 GPUs from 9 hours to 5 minutes.
Paper Structure (38 sections, 3 theorems, 37 equations, 14 figures, 1 table, 2 algorithms)

This paper contains 38 sections, 3 theorems, 37 equations, 14 figures, 1 table, 2 algorithms.

Table of Contents

  1. Introduction
  2. Background and Motivation
  3. Network Simulation in LLM Training
  4. Repeated Flow Patterns in LLM Training
  5. Steady-state in LLM Training
  6. Wormhole Overview
  7. Terminology & Formal Definitions
  8. Port-level Network Partition
  9. Steady-state
  10. Workflow
  11. Memoization for Unsteady-states
  12. Network Partitioning Algorithm
  13. Flow Conflict Graph
  14. Storing Unsteady-states
  15. Retrieving and Skipping Unsteady-states
  16. ...and 23 more sections

Key Result

Theorem 1

Within the time interval [$t_s$, $t_f$], the congestion control algorithm converges. Assuming the flow rate is stable, i.e., it satisfies $\Delta R < \epsilon_R$, then for $cwnd$, RTT, $Q$, and $I$, they are also stable. This means when Equation formula:r_tot_stable holds, there exist small constants $\epsilon_{cwnd}$, $\epsilon_{RTT}$, $\epsilon_Q$, and $\epsilon_I$ such that the following in

Figures (14)

  • Figure 1: Illustrating unsteady-states, steady-states, state memoization, and simulation fast-forwarding in packet-level network simulation at the scenario of LLM training.
  • Figure 2: The efficiency and accuracy of existing network simulation techniques for LLM training.
  • Figure 3: Repeated contention patterns and steady-states in LLM training.
  • Figure 4: Examples of network partitions and the corresponding Flow Conflict Graphs (FCGs).
  • Figure 5: Flow sending rate during CCA convergence.
  • ...and 9 more figures

Theorems & Definitions (8)

  • Definition 1: Network Partition
  • Definition 2: Traffic Steady-state
  • Theorem 1
  • Theorem 2
  • Theorem 3
  • proof
  • proof
  • proof