Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Teola: Towards End-to-End Optimization of LLM-based Applications

Xin Tan, Yimin Jiang, Yitao Yang, Hong Xu

TL;DR

Teola tackles end-to-end latency in LLM-based applications by replacing coarse module chains with primitive-level dataflow graphs. It introduces a Graph Optimizer that generates per-query p-graphs and an execution engine with a two-tier Runtime Scheduler, enabling parallelization, pipelining, and application-aware batching. Empirical results show up to 2.09x end-to-end speedups across diverse workflows and co-location scenarios. The work demonstrates that finer-grained orchestration across LLM and non-LLM components can substantially close the gap between per-module latency and user-perceived performance, with practical implications for production AI systems.

Abstract

Large language model (LLM)-based applications consist of both LLM and non-LLM components, each contributing to the end-to-end latency. Despite great efforts to optimize LLM inference, end-to-end workflow optimization has been overlooked. Existing frameworks employ coarse-grained orchestration with task modules, which confines optimizations to within each module and yields suboptimal scheduling decisions. We propose fine-grained end-to-end orchestration, which utilizes task primitives as the basic units and represents each query's workflow as a primitive-level dataflow graph. This explicitly exposes a much larger design space, enables optimizations in parallelization and pipelining across primitives of different modules, and enhances scheduling to improve application-level performance. We build Teola, a novel orchestration framework for LLM-based applications that implements this scheme. Comprehensive experiments show that Teola can achieve up to 2.09x speedup over existing systems across various popular LLM applications. The code is available at https://github.com/NetX-lab/Ayo.

Teola: Towards End-to-End Optimization of LLM-based Applications

TL;DR

Teola tackles end-to-end latency in LLM-based applications by replacing coarse module chains with primitive-level dataflow graphs. It introduces a Graph Optimizer that generates per-query p-graphs and an execution engine with a two-tier Runtime Scheduler, enabling parallelization, pipelining, and application-aware batching. Empirical results show up to 2.09x end-to-end speedups across diverse workflows and co-location scenarios. The work demonstrates that finer-grained orchestration across LLM and non-LLM components can substantially close the gap between per-module latency and user-perceived performance, with practical implications for production AI systems.

Abstract

Large language model (LLM)-based applications consist of both LLM and non-LLM components, each contributing to the end-to-end latency. Despite great efforts to optimize LLM inference, end-to-end workflow optimization has been overlooked. Existing frameworks employ coarse-grained orchestration with task modules, which confines optimizations to within each module and yields suboptimal scheduling decisions. We propose fine-grained end-to-end orchestration, which utilizes task primitives as the basic units and represents each query's workflow as a primitive-level dataflow graph. This explicitly exposes a much larger design space, enables optimizations in parallelization and pipelining across primitives of different modules, and enhances scheduling to improve application-level performance. We build Teola, a novel orchestration framework for LLM-based applications that implements this scheme. Comprehensive experiments show that Teola can achieve up to 2.09x speedup over existing systems across various popular LLM applications. The code is available at https://github.com/NetX-lab/Ayo.
Paper Structure (24 sections, 13 figures, 3 tables, 2 algorithms)

This paper contains 24 sections, 13 figures, 3 tables, 2 algorithms.

Table of Contents

  1. Introduction
  2. Background and Motivation
  3. LLM-based Applications
  4. Fine-grained Orchestration of LLM Apps
  5. Application-Aware Scheduling and Execution
  6. Design Overview
  7. Architecture
  8. APIs
  9. Graph Optimizer
  10. p-Graph
  11. Optimization
  12. Runtime Scheduling
  13. Graph Scheduler
  14. Engine Scheduler
  15. Co-located Applications
  16. ...and 9 more sections

Figures (13)

  • Figure 1: Latency breakdown of each task module for various applications in Figure \ref{['fig:llm-apps']} using LlamaIndex llamaindex. The LLM synthesizing module time is divided into prefilling and decoding.
  • Figure 2: Common real-world LLM-based application workflows obtained from orchestration frameworks pairagllamaindexlangchain and Anthropic contextual-retrieval.
  • Figure 3:
  • Figure 7: Batching for embedding engine.
  • Figure 8: Batching for LLM engine.
  • ...and 8 more figures