Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Thinking Short and Right Over Thinking Long: Serving LLM Reasoning Efficiently and Accurately

Yuhang Wang, Youhe Jiang, Bin Cui, Fangcheng Fu

TL;DR

This work targets the efficiency-accuracy trade-off in LLM reasoning when combining Chain-of-Thought-like sequential steps with parallel branch sampling. It introduces SART, a serving framework that enforces short, high-quality thinking by employing redundant sampling with early stopping and a two-phase dynamic pruning strategy guided by a process reward model, enabling continuous batching. Empirically, SART achieves substantial speedups (up to $28.2\times$, average $15.7\times$) at comparable accuracy across multiple models and datasets, while reducing queuing delays and resource consumption. The approach enables practical, scalable deployment of reasoning-enabled LLMs, balancing latency and quality in multi-request serving scenarios.

Abstract

Recent advances in test-time scaling suggest that Large Language Models (LLMs) can gain better capabilities by generating Chain-of-Thought reasoning (analogous to human thinking) to respond a given request, and meanwhile exploring more reasoning branches (i.e., generating multiple responses and ensembling them) can improve the final output quality. However, when incorporating the two scaling dimensions, we find that the system efficiency is dampened significantly for two reasons. Firstly, the time cost to generate the final output increases substantially as many reasoning branches would be trapped in the over-thinking dilemma, producing excessively long responses. Secondly, generating multiple reasoning branches for each request increases memory consumption, which is unsuitable for LLM serving since we can only batch a limited number of requests to process simultaneously. To address this, we present SART, a serving framework for efficient and accurate LLM reasoning. The essential idea is to manage the thinking to be short and right, rather than long. For one thing, we devise a redundant sampling with early stopping approach based on empirical observations and theoretic analysis, which increases the likelihood of obtaining short-thinking responses when sampling reasoning branches. For another, we propose to dynamically prune low-quality branches so that only right-thinking branches are maintained, reducing the memory consumption and allowing us to batch more requests. Experimental results demonstrate that SART not only improves the accuracy of LLM reasoning but also enhances the serving efficiency, outperforming existing methods by up to 28.2 times and on average 15.7 times in terms of efficiency when achieving the same level of accuracy.

Thinking Short and Right Over Thinking Long: Serving LLM Reasoning Efficiently and Accurately

TL;DR

This work targets the efficiency-accuracy trade-off in LLM reasoning when combining Chain-of-Thought-like sequential steps with parallel branch sampling. It introduces SART, a serving framework that enforces short, high-quality thinking by employing redundant sampling with early stopping and a two-phase dynamic pruning strategy guided by a process reward model, enabling continuous batching. Empirically, SART achieves substantial speedups (up to , average ) at comparable accuracy across multiple models and datasets, while reducing queuing delays and resource consumption. The approach enables practical, scalable deployment of reasoning-enabled LLMs, balancing latency and quality in multi-request serving scenarios.

Abstract

Recent advances in test-time scaling suggest that Large Language Models (LLMs) can gain better capabilities by generating Chain-of-Thought reasoning (analogous to human thinking) to respond a given request, and meanwhile exploring more reasoning branches (i.e., generating multiple responses and ensembling them) can improve the final output quality. However, when incorporating the two scaling dimensions, we find that the system efficiency is dampened significantly for two reasons. Firstly, the time cost to generate the final output increases substantially as many reasoning branches would be trapped in the over-thinking dilemma, producing excessively long responses. Secondly, generating multiple reasoning branches for each request increases memory consumption, which is unsuitable for LLM serving since we can only batch a limited number of requests to process simultaneously. To address this, we present SART, a serving framework for efficient and accurate LLM reasoning. The essential idea is to manage the thinking to be short and right, rather than long. For one thing, we devise a redundant sampling with early stopping approach based on empirical observations and theoretic analysis, which increases the likelihood of obtaining short-thinking responses when sampling reasoning branches. For another, we propose to dynamically prune low-quality branches so that only right-thinking branches are maintained, reducing the memory consumption and allowing us to batch more requests. Experimental results demonstrate that SART not only improves the accuracy of LLM reasoning but also enhances the serving efficiency, outperforming existing methods by up to 28.2 times and on average 15.7 times in terms of efficiency when achieving the same level of accuracy.
Paper Structure (10 sections, 1 theorem, 7 figures, 1 algorithm)

This paper contains 10 sections, 1 theorem, 7 figures, 1 algorithm.

Table of Contents

  1. Introduction
  2. Background and Related Works
  3. Motivation and Analysis
  4. Method
  5. Evaluation
  6. Implementation and Experimental Setup
  7. End-to-end Comparison
  8. Ablation Studies
  9. Sensitivity Analysis
  10. Conclusion and Limitations

Key Result

Lemma 1

Let $X_1, X_2, \cdots, X_N$ be random variables sampled from a cumulative distribution of $F_X(x)$. The $M$-th smallest value, denoted as $X_{(M)}$, has cumulative distribution $F_{X_{(M)}}(x;N) = \sum_{i=M}^N {N \choose i} [F_X(x)]^i[1 - F_X(x)]^{N-i}$.

Figures (7)

  • Figure 1: Illustration of the two key challenges and our solutions.
  • Figure 2: The numbers of correct and wrong responses in each length range, testing with three questions respectively. Each $i$-$j$ on $x$-axis denotes a range of response lengths (e.g., "7-8" indicates 7K-8K tokens).
  • Figure 3: The numbers of running branches and tokens with and without pruning. The redundant sampling with early stopping is enabled ($N=8, M=4$).
  • Figure 4: An example of our two-phase pruning. Each value represents the reward of each branch at the corresponding decoding step.
  • Figure 5: End-to-end (E2E) latency (lower is better) and accuracy (higher is better) of each method with different $N$. We plot a horizontal line for Vanilla (which corresponds to $N=1$) to indicate the baseline performance of reasoning without branch sampling.
  • ...and 2 more figures

Theorems & Definitions (1)

  • Lemma 1