Think Deep, Think Fast: Investigating Efficiency of Verifier-free Inference-time-scaling Methods

TL;DR

The paper investigates verifier-free inference-time scaling (ITC) for large language models, comparing reasoning-specialized models with general non-reasoning models across challenging benchmarks. By constructing Pareto frontiers of quality versus efficiency, it shows that majority voting is a robust, cost-effective ITC baseline, while more complex methods yield limited gains for reasoning models and do not salvage non-reasoning models. It further analyzes how response length and linguistic markers correlate with correctness, revealing that shorter, less hedged responses tend to be more accurate in reasoning models and that markers can serve as useful predictors of output quality. The results advocate prioritizing the development and deployment of reasoning-focused models and propose leveraging linguistic signals to refine ITC strategies without increasing computation.

Abstract

There is intense interest in investigating how inference time compute (ITC) (e.g. repeated sampling, refinements, etc) can improve large language model (LLM) capabilities. At the same time, recent breakthroughs in reasoning models, such as Deepseek-R1, unlock the opportunity for reinforcement learning to improve LLM reasoning skills. An in-depth understanding of how ITC interacts with reasoning across different models could provide important guidance on how to further advance the LLM frontier. This work conducts a comprehensive analysis of inference-time scaling methods for both reasoning and non-reasoning models on challenging reasoning tasks. Specifically, we focus our research on verifier-free inference time-scaling methods due to its generalizability without needing a reward model. We construct the Pareto frontier of quality and efficiency. We find that non-reasoning models, even with an extremely high inference budget, still fall substantially behind reasoning models. For reasoning models, majority voting proves to be a robust inference strategy, generally competitive or outperforming other more sophisticated ITC methods like best-of-N and sequential revisions, while the additional inference compute offers minimal improvements. We further perform in-depth analyses of the association of key response features (length and linguistic markers) with response quality, with which we can improve the existing ITC methods. We find that correct responses from reasoning models are typically shorter and have fewer hedging and thinking markers (but more discourse markers) than the incorrect responses.

Figures (6)

• Figure 1: The overview of inference-time-compute methods for reasoning and non-reasoning models. Even though inference-time scaling method improves Llama-3.3-70B, it still struggles to beat the R1-distilled version of Llama 70B. However, with very limited compute, non-reasoning model with inference method can be at the pareto front.
• Figure 2: Performance of various inference-time scaling methods for four models across MATH, AIME and GPQA. For some methods we have multiple eval metrics. For bon (best-of-N approach), we pick the highest scored response. majority topk means we use top k scored resposnes to do majority voting (we always set k as half of total samples). last means we pick the last revisioned sample. chain best majority indicates that we use the best scored sample from each chain and then take majority. Due to high cost of inference, methods like sequential revisions and combined sequential parallel are only sampled once, which may seem volatile when plotted. The results for other models can be found in \ref{['appendix:more_inf_results']}.
• Figure 3: The average response length gap for each model tasks across four tasks. The average response length gap is computed by: 1) calculating mean length difference between correct and incorrect responses within each question and 2) averaging these differences across the entire dataset. LCB_CODEGEN represents the code generation subtask in the LiveCodeBench benchmark.
• Figure 4: Accuracy of responses in different length groups. For each question, we generate 100 samples and then we bin those samples into five bins. Then average accuracy is computed for each bin across the dataset.
• Figure 5: Average gaps between correct and incorrect responses for DeepSeek-R1-Distill-Llama-70B and DeepSeek-R1-Distill-Llama-14B. The average gaps are first computed by using computing the mean difference of thinking token frequency of correct and incorrect responses within one question and then average over the entire dataset. The frequency is weighted by response length. Refer to \ref{['tab:linguistic_markers']} for the definition of different marker categories.