Breaking the Overscaling Curse: Thinking Parallelism Before Parallel Thinking

TL;DR

The paper addresses inefficiency in test-time parallel thinking caused by sample heterogeneity that triggers the overscaling curse, where maximizing dataset accuracy with a global budget $N_{\mathcal{D}}$ wastes computation on many samples. It introduces T2, a method that predicts per-sample optimal parallelism using latent representations from multiple transformer layers and an estimator-weighted aggregation to compute $\hat{N}_{\bm x}^*$. Empirical results show T2 substantially reduces memory and latency while preserving accuracy across diverse models and datasets, including OOD settings, by effectively allocating budgets before decoding. This per-sample, latent-budget approach enables more efficient parallel reasoning and broader applicability of test-time parallelism without sacrificing performance.

Abstract

Parallel thinking enhances LLM reasoning by multi-path sampling and aggregation. In system-level evaluations, a global parallelism level N is allocated to all samples, typically set large to maximize overall dataset accuracy. However, due to sample heterogeneity, some samples can achieve comparable performance with a smaller N'< N, causing budget redundancy. This incompatibility between system-level efficacy and sample-level efficiency constitutes the overscaling curse. In this paper, we formalize and quantify the overscaling curse, showing its universality and severity in practice, and analyze its trigger mechanism. We then propose a lightweight method, T2, to break the overscaling curse, which utilizes latent representations to estimate the optimal parallelism level for each sample before decoding. Experiments show that T2 significantly reduces cost while maintaining comparable performance, enabling more efficient parallel thinking.

Figures (24)

• Figure 1: The Overscaling Curse of Parallel Thinking. When evaluating an entire dataset $\mathcal{D}$ in a model, a large global parallelism level $N_{\mathcal{D}}$ is often used to maximize overall performance, as in Episode (ii). Under this, as in Episode (i), only type-(4) samples truly benefit, since they indeed require large $N$ to realize substantial gains. In contrast, the other sample types do not benefit because they reach their best performance with only a small $N$. Therefore, maximizing system-level efficacy usually introduces budget redundancy for some individual samples, i.e., reducing sample-level efficiency. Examples here are from Qwen3-4B on the AIME25 dataset, with more shown in Appendix \ref{['appe:more-type-examples']} (Figure \ref{['fig:overscaling-sample-qwen25-math-type3']} - \ref{['fig:overscaling-sample-qwen3-aime25-type5']}).
• Figure 2: T2: Thinking Parallelism Before Parallel Thinking. We introduce trainable layer-wise estimators that predict the optimal parallelism level for each input from its final-token representations. These estimators are first trained, and each is assigned a weight based on its layer-wise validation error. During inference, after encoding the input, the layer-weighted parallelism estimate $\hat{N}^*$ is obtained, and then decoding is performed under this budget. T2 effectively breaks the overscaling curse and enables more efficient parallel thinking.
• Figure 3: OverScaling Index$\mathcal{M}_{\mathcal{D}}$ across models and datasets, with detailed ($N_{\mathcal{D}}^* / N_{\mathcal{D}}$) labeled below each value.
• Figure 4: Proportion of the five sample types across datasets in Qwen3-4B. Results of other models are shown in Appendix \ref{['appe:more-proportion']}.
• Figure 5: Estimation Results of Layer-wise Estimators. Each estimator is trained over 8 runs. Points indicate the mean, while the shaded areas indicate the standard deviation. Datasets with blue lines denote in-domain, and red lines denote out-of-domain datasets.

Theorems & Definitions (7)

• Definition 2.1
• Definition 3.1: Overscaling Curse
• Definition 3.2: Overscaling Index $\mathcal{M}_{\mathcal{D}}$
• Definition 3.3: Approximate Monotonicity of $A_{\bm{x}}(N)$
• Theorem 3.4
• Theorem 4.1
• proof