BeLLMan: Controlling LLM Congestion

TL;DR

Problem: LLM applications are often oblivious to system load, causing latency spikes during congestion. Approach: beLLMan introduces a first-cut, prompt-based congestion controller that binds output length by instructing the LLM to generate 'in exactly $N$ words', where $N$ is derived from the unbounded length $L$ and a tunable reduction $r$ determined by congestion signals. Key results: on Gemma-$3$ (27B) running on a DGX with 8×$H100$ GPUs, end-to-end latency improves up to $8\times$, energy usage drops by about $25\%$, and $19\%$ more requests complete, with similarity to unbounded outputs remaining high (≈$87\%$ vs $88\%$). Significance: demonstrates a scalable, software-only method to alleviate LLM congestion without retraining or swapping models, with potential energy and throughput benefits at scale.

Abstract

Large language model (LLM) applications are blindfolded to the infrastructure underneath and generate tokens autoregressively, indifferent to the system load, thus risking inferencing latency inflation and poor user experience. Our first-cut controller, named beLLMan, enables the LLM infrastructure to actively and progressively signal the first-party LLM application to adjust the output length in response to changing system load. On a real testbed with H100 GPUs, beLLMan helps keep inferencing latency under control (upto 8X lower end-to-end latency) and reduces energy consumption by 25% (while serving 19% more requests) during periods of congestion for a summarization workload.

Figures (4)

• Figure 1: Internet (top) versus LLM congestion control.
• Figure 2: LLM behavior under different configurations.
• Figure 3: beLLMan architecture diagram.
• Figure 4: Synthetic trace and the corresponding unbounded and bounded runs.