Input-Time Scaling: Adding Noise and Irrelevance into Less-Is-More Drastically Improves Reasoning Performance and Efficiency

TL;DR

The paper challenges the necessity of high-quality data for strong reasoning in LLMs by introducing train-test co-design and Input-Time Scaling. By injecting controlled noise through persona contexts and systematically comparing high- and low-quality datasets, it reveals that consistency of context augmentation across training and testing, including irrelevant noise, can enhance reasoning efficiency and performance even with very small data (around 1K examples). The approach yields state-of-the-art results among 32B models on AIME tasks and demonstrates a capacity-dependent quality-capacity tradeoff, where low-quality data can outperform high-quality data as models scale. The work offers practical implications for reducing data curation costs while maintaining or improving reasoning effectiveness, and it provides open-source resources to reproduce and extend these findings.

Abstract

Large Language Models (LLMs) excel at reasoning, traditionally requiring high-quality large-scale data and extensive training. Recent works reveal a very appealing Less-Is-More phenomenon where very small, carefully curated high-quality datasets match resource-intensive approaches. In this work, we further systematically relax their quality constraints by adding controlled noise via persona context relevance and comparing datasets of different qualities. Counterintuitively, we find that mixing relevant and irrelevant contexts consistently across training and inference stages yields optimal results -- a phenomenon we term training-testing co-design. Dataset quality comparisons show that high-quality data benefits weaker models on easy questions, while low-quality data achieves higher scores on hard questions with capable models. Across our experiments, reasoning performance is linked to reasoning efficiency. We, for the first time, found adding noisy and irrelevant contexts into queries can improve reasoning efficiency without any prices and targeted designs. Building on these insights, we propose Input-Time Scaling: applying small, low-quality data to capable models with training-testing co-design. This maintains Less-Is-More while further removing labor-intensive quality curation and improving reasoning effectiveness and efficiency, making the approach more applicable and affordable. Our method achieves 76.7% pass@1 on AIME24/25 using Qwen2.5-32B-Instruct, and 90.0%/80.0% with DeepSeek-R1-Distill-Qwen-32B -- state-of-the-art among Qwen2.5-32B variants. We are open-sourcing our datasets, pipelines, evaluation results, and checkpoints to facilitate reproducibility and further research.

Figures (4)

• Figure 1: Data Processing Pipeline Overview: (a) Single datapoint processing: we use meta-cognition to generate a persona based on the input, concatenate it to the original query, and retain the original CoTs & Answers. The strategy determines persona type (see Section \ref{['methods']}). (b) Dataset construction: we apply the same strategy to all datapoints, creating four datasets with different persona types (including one unchanged baseline under strategy-N). Persona-query relevance controls noise level.
• Figure 2: Performance Comparison Between Training and Testing Strategies of Qwen2.5-32B-Instruct: (a,c) are the training visualizations ; (b,d) are the testing visualizations. Each sub figure contains two part and both are compared to -N strategy. Left contains the output length change rates, the right is the visulizations into thinking categories. Applying any strategy during training or testing will shorten the output length, making the reasoning more efficient, which is linked to improved performance.
• Figure 3: Performance comparison between training and testing strategies of DeepSeek-R1-Distill-qwen-32B: (a,c) is the training visualization; (b,d) is the testing visualization.
• Figure 4: Performance comparison between training strategies