Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Improving the End-to-End Efficiency of Offline Inference for Multi-LLM Applications Based on Sampling and Simulation

Jingzhi Fang, Yanyan Shen, Yue Wang, Lei Chen

TL;DR

This work tackles offline end-to-end throughput for multi-LLM applications on a single-node, multi-GPU platform by jointly scheduling which models run concurrently and selecting their parallelism plans. It introduces a sampling-then-simulation cost model to estimate LLM run times under unknown output lengths, and formalizes the problem as a NP-hard computation-graph scheduling task, solved via a greedy search to generate execution stages. The SamuLLM framework then executes the planned stages with a dynamic running phase that adapts to real-time runtime signals, enabling model preemption and reallocation guided by hardware constraints such as NV-link connectivity. Empirical results across multiple multi-LLM workloads show end-to-end speedups of 1.0–2.4× compared to heuristics, validating the effectiveness of planning-based throughput optimization for offline, multi-LLM inference.

Abstract

As large language models (LLMs) have shown great success in many tasks, they are used in various applications. While a lot of works have focused on the efficiency of single-LLM application (e.g., offloading, request scheduling, parallelism strategy selection), multi-LLM applications receive less attention, particularly in offline inference scenarios. In this work, we aim to improve the offline end-to-end inference efficiency of multi-LLM applications in the single-node multi-GPU environment. The problem involves two key decisions: (1) determining which LLMs to run concurrently each time (we may not run all the models at the same time), and (2) selecting a parallelism strategy to use for each LLM. This problem is NP-hard. Naive solutions may not work well because the running time for a model to complete a set of requests depends on the request workload and the selected parallelism strategy, and they lack an accurate model of the running time. As the LLM output lengths are unknown before running, to estimate the model running time, we propose a sampling-then-simulation method which first estimates the output lengths by sampling from an empirical cumulative function we obtained from a large dataset in advance, and then simulates the LLM inference process accordingly. Based on the simulation, we estimate the per-iteration latencys to get the total latency. A greedy method is proposed to optimize the scheduling of the LLMs in the application across the GPUs. We then propose a framework SamuLLM which contains two phases: planning, which calls the greedy method for an application and running, which runs the application and dynamically adjust the model scheduling based on the runtime information. Experiments on 3 applications and a mixed application show that SamuLLM can achieve 1.0-2.4$\times$ end-to-end speedups compared to the competitors.

Improving the End-to-End Efficiency of Offline Inference for Multi-LLM Applications Based on Sampling and Simulation

TL;DR

This work tackles offline end-to-end throughput for multi-LLM applications on a single-node, multi-GPU platform by jointly scheduling which models run concurrently and selecting their parallelism plans. It introduces a sampling-then-simulation cost model to estimate LLM run times under unknown output lengths, and formalizes the problem as a NP-hard computation-graph scheduling task, solved via a greedy search to generate execution stages. The SamuLLM framework then executes the planned stages with a dynamic running phase that adapts to real-time runtime signals, enabling model preemption and reallocation guided by hardware constraints such as NV-link connectivity. Empirical results across multiple multi-LLM workloads show end-to-end speedups of 1.0–2.4× compared to heuristics, validating the effectiveness of planning-based throughput optimization for offline, multi-LLM inference.

Abstract

As large language models (LLMs) have shown great success in many tasks, they are used in various applications. While a lot of works have focused on the efficiency of single-LLM application (e.g., offloading, request scheduling, parallelism strategy selection), multi-LLM applications receive less attention, particularly in offline inference scenarios. In this work, we aim to improve the offline end-to-end inference efficiency of multi-LLM applications in the single-node multi-GPU environment. The problem involves two key decisions: (1) determining which LLMs to run concurrently each time (we may not run all the models at the same time), and (2) selecting a parallelism strategy to use for each LLM. This problem is NP-hard. Naive solutions may not work well because the running time for a model to complete a set of requests depends on the request workload and the selected parallelism strategy, and they lack an accurate model of the running time. As the LLM output lengths are unknown before running, to estimate the model running time, we propose a sampling-then-simulation method which first estimates the output lengths by sampling from an empirical cumulative function we obtained from a large dataset in advance, and then simulates the LLM inference process accordingly. Based on the simulation, we estimate the per-iteration latencys to get the total latency. A greedy method is proposed to optimize the scheduling of the LLMs in the application across the GPUs. We then propose a framework SamuLLM which contains two phases: planning, which calls the greedy method for an application and running, which runs the application and dynamically adjust the model scheduling based on the runtime information. Experiments on 3 applications and a mixed application show that SamuLLM can achieve 1.0-2.4 end-to-end speedups compared to the competitors.

Paper Structure

This paper contains 15 sections, 1 theorem, 5 equations, 15 figures, 1 table, 1 algorithm.

Table of Contents

  1. Introduction
  2. Cost Model Insight
  3. Problem Formulation
  4. Methodology
  5. Cost model
  6. Search Method
  7. Running phase
  8. Experiments
  9. LLM ensembling
  10. LLM routing
  11. Chain summary
  12. Mixed application
  13. Ablation study
  14. Related Work
  15. Conclusion

Key Result

Theorem 1

The application plan search problem is NP-hard.

Figures (15)

  • Figure 1: Different application running plans result in different end-to-end efficiency. The running time of a model does not necessarily decrease linearly with the number of GPUs assigned to it. This figure shows 4 different execution plans for 6 LLMs in an application with no dependency among the models: (a) running LLMs sequentially and assigning all GPUs to the running LLM each time; (b) running as many LLMs concurrently as possible and splitting the GPUs among the LLMs event (model preemption is not allowed); (c) the same as (b) but allowing model preemption; (d) assigning more GPUs to model 1 in the second stage to achieve higher throughput, rather than always partitioning the GPUs evenly.
  • Figure 2: Output length eCDFs: (a) eCDFs of different input length regions; (b) eCDFs of different request categories; (c) eCDFs of different input length regions in the "Rewrite" category.
  • Figure 3: Running request number of each iteration in (a) the real inference process and (b) the simulated inference process, respectively.
  • Figure 4: Three major components of the LLM per-iteration latency: (a) model computation latency, (b) input preparation latency, and (c) output token sampling latency.
  • Figure 5: The computation graphs of different applications.
  • ...and 10 more figures

Theorems & Definitions (2)

  • Definition 1: Application Plan Search Problem
  • Theorem 1