Prompting Test-Time Scaling Is A Strong LLM Reasoning Data Augmentation

TL;DR

This work tackles data-efficient elicitation of LLM reasoning by introducing Prompting Test-Time Scaling (P-TTS), which converts a small seed pool of 90 high-quality math problems into a large, diverse supervision signal through principled instruction wrappers and test-time prompt ensembles. By capturing multiple reasoning trajectories via core prompts (Reward, Penalty, Correctness, StepByStep) and paraphrased rewards, and by generating teacher-produced reasoning traces, P-TTS enables fine-tuning of models such as Qwen2.5-Instruct across sizes with far fewer annotated examples. Across AIME, MATH500, GPQA-Diamond, and cross-domain benchmarks, P-TTS demonstrates substantial gains over strong baselines and shows robust zero-shot generalization, confirming that prompt-space exploration can effectively scale reasoning without massive labeled data. The method offers a practical, low-cost pathway for building reasoning capabilities in resource-constrained or rapidly shifting domains and suggests future extensions to adaptive wrappers, retrieval-grounded setups, and curriculum-like scheduling of prompts.

Abstract

Large language models (LLMs) have demonstrated impressive reasoning capabilities when provided with chain-of-thought exemplars, but curating large reasoning datasets remains laborious and resource-intensive. In this work, we introduce Prompting Test-Time Scaling (P-TTS), a simple yet effective inference-time data augmentation strategy for enhancing LLM reasoning through finetuning. Rather than collecting thousands or even millions of examples, P-TTS leverages a small pool of only 90 manually selected reasoning instances and systematically varies exemplar augmentation through principled instruction prompting intensities at test time to synthesize diverse reasoning trajectory contexts. Then we finetune the various sizes of Qwen-2.5 models on P-TTS data. Across a suite of mathematical reasoning AIME2024 & 25, MATH500, and GPQA-Diamond, our P-TTS-7B and 32B models outperform the prior competitive baselines like S1 and S1.1 (1K-shot), achieving absolute accuracy gains of +26.66% and +30.00% on AIME'24 (7B), and +13.34% and +6.67% on AIME'25 (7B); P-TTS-32B yields gains of +23.33% and +16.63% on AIME'24, and +26.63% and +3.33% on AIME'25 (vs. S1 and S1.1, respectively), with comparable or better performance on MATH500 and GPQA-Diamond. We further show that P-TTS enhances zero-shot generalization accuracy on out-of-domain reasoning benchmarks of Gaokao, Kaoyan, OlympiadBench, AMC23, GradeSchoolMath, and Minerva. Our analysis suggests that test-time scaling effectively explores the latent space of reasoning patterns, amplifying LLM problem-solving with minimal annotation overhead, and further unlocking the reasoning potential and capabilities of LLMs. Prompting Test-Time Scaling offers a practical, low-cost way to elicit LLM reasoning in resource-constrained or rapidly evolving domains.

Figures (9)

• Figure 1: Comparison of 32B-scale models on AIME 2024 (left), MATH500 (middle), and GPQA Diamond (right). Model performance on AIME 2024, MATH500, and GPQA-Diamond benchmarks as a function of dataset size. Each point represents a different model, with our P-TTS-32B (red star) showing competitive performance from a significantly smaller dataset. The x-axis scale highlights the differences in training data sizes across models.
• Figure 2: Overview of the P-TTS data augmentation process. Starting from a small set of high-quality math problems (AIME-style), we generate diverse prompt variants through instruction reframing, such as reward-based encouragement, penalty warnings, and step-by-step guidance. These augmented prompts are used to elicit high-quality LLM completions, which are then collected as synthetic reasoning data to fine-tune.
• Figure 3: Accuracy improvement with increased principled augmentation on 7B model. We evaluate how model accuracy scales with the number of augmented training examples. Here, $\times$1 refers to P-TTSReward (90 examples), $\times$4 to P-TTSCore (360 examples), $\times$5 to P-TTSCore+Orig (450 examples), and $\times$10 to P-TTSFull (900 examples). Accuracy improves consistently across all evaluation sets with larger, principle-guided augmentations.
• Figure 4: Knowledge Diversity Gain vs. Accuracy for different P-TTS variants across four benchmarks. We compare the trade-off between Accuracy (blue solid line, left y-axis) and Knowledge Diversity Gain (gray dashed line, right y-axis) on 7B model for four principled prompting strategies: Reward, Correctness, Penalty, and Think. Diversity Gain is computed relative to the original P-TTS baseline.
• Figure 5: Impact of incremental principle addition on average accuracy. As additional prompting principles are cumulatively incorporated into training (P6 → +P10 → +P12 → +P9), both the number of training examples and model accuracy increase. Bars (right axis) denote the total number of examples after each addition; the green line (left axis) shows the resulting average accuracy across evaluation benchmarks. This highlights the compounding benefit of principled augmentation.