Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Parallel Track Transformers: Enabling Fast GPU Inference with Reduced Synchronization

Chong Wang, Nan Du, Tom Gunter, Tao Lei, Kulin Seth, Senyu Tong, Jianyu Wang, Guoli Yin, Xiyou Zhou, Kelvin Zou, Ruoming Pang

TL;DR

The paper tackles the synchronization bottleneck in tensor-parallel LLM inference by introducing the Parallel Track (PT) Transformer, which partitions a model into multiple independent tracks that periodically fuse activations. By adjusting track depth $D$, PT reduces synchronization points from $2L$ to $L/D$ and lowers exchange data, enabling faster serving without sacrificing model quality, as demonstrated across 6B, 13B, and 30B models. Experiments in TensorRT-LLM and vLLM show PT achieves 15–30% faster time-to-first-token, 2–12% faster time-per-output-token, and up to 31.9% higher throughput, with larger models tolerating higher $D$ with minimal degradation. The work also discusses a PT-MoE extension, combining track-based synchronization with mixture-of-experts within tracks, positioning PT as a practical approach for scalable, low-latency LLM serving in private clouds and production stacks.

Abstract

Efficient large-scale inference of transformer-based large language models (LLMs) remains a fundamental systems challenge, frequently requiring multi-GPU parallelism to meet stringent latency and throughput targets. Conventional tensor parallelism decomposes matrix operations across devices but introduces substantial inter-GPU synchronization, leading to communication bottlenecks and degraded scalability. We propose the Parallel Track (PT) Transformer, a novel architectural paradigm that restructures computation to minimize cross-device dependencies. PT achieves up to a 16x reduction in synchronization operations relative to standard tensor parallelism, while maintaining competitive model quality in our experiments. We integrate PT into two widely adopted LLM serving stacks-Tensor-RT-LLM and vLLM-and report consistent improvements in serving efficiency, including up to 15-30% reduced time to first token, 2-12% reduced time per output token, and up to 31.90% increased throughput in both settings.

Parallel Track Transformers: Enabling Fast GPU Inference with Reduced Synchronization

TL;DR

The paper tackles the synchronization bottleneck in tensor-parallel LLM inference by introducing the Parallel Track (PT) Transformer, which partitions a model into multiple independent tracks that periodically fuse activations. By adjusting track depth , PT reduces synchronization points from to and lowers exchange data, enabling faster serving without sacrificing model quality, as demonstrated across 6B, 13B, and 30B models. Experiments in TensorRT-LLM and vLLM show PT achieves 15–30% faster time-to-first-token, 2–12% faster time-per-output-token, and up to 31.9% higher throughput, with larger models tolerating higher with minimal degradation. The work also discusses a PT-MoE extension, combining track-based synchronization with mixture-of-experts within tracks, positioning PT as a practical approach for scalable, low-latency LLM serving in private clouds and production stacks.

Abstract

Efficient large-scale inference of transformer-based large language models (LLMs) remains a fundamental systems challenge, frequently requiring multi-GPU parallelism to meet stringent latency and throughput targets. Conventional tensor parallelism decomposes matrix operations across devices but introduces substantial inter-GPU synchronization, leading to communication bottlenecks and degraded scalability. We propose the Parallel Track (PT) Transformer, a novel architectural paradigm that restructures computation to minimize cross-device dependencies. PT achieves up to a 16x reduction in synchronization operations relative to standard tensor parallelism, while maintaining competitive model quality in our experiments. We integrate PT into two widely adopted LLM serving stacks-Tensor-RT-LLM and vLLM-and report consistent improvements in serving efficiency, including up to 15-30% reduced time to first token, 2-12% reduced time per output token, and up to 31.90% increased throughput in both settings.
Paper Structure (13 sections, 1 figure, 10 tables, 1 algorithm)

This paper contains 13 sections, 1 figure, 10 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. The Parallel Track Transformer Architecture
  3. Background
  4. PT Transformer
  5. Independent branches with periodic fusion.
  6. Contrast with MoE / expert parallelism.
  7. Experiments
  8. Model Configurations
  9. Model Performance Comparison
  10. Serving Evaluation
  11. TensorRT-LLM
  12. vLLM
  13. Conclusion

Figures (1)

  • Figure 1: Diagram of the Parallel Track (PT) transformer architecture. Each track is composed of multiple track blocks, and each track block contains a fixed number of standard transformer layers, defined by the block depth.