T1: Tool-integrated Self-verification for Test-time Compute Scaling in Small Language Models

TL;DR

This work tackles the challenge of self-verification in small language models under test-time compute scaling, where reliance on large verifiers erodes efficiency. It introduces Tool-integrated Self-verification (T1), a two-stage framework that first filters outputs via external tools (ToolV) and then scores remaining candidates with a reward-model verifier, with distillation from large teachers to boost both stages. Theoretical results show that tool use dramatically reduces memorization demands, converting verification into a learnable task for sLMs, thereby improving scaling performance even with imperfect verifiers. Empirically, ToolV yields substantial gains on math benchmarks (MATH500, GSM8K) and multi-domain tasks (MMLU-Pro), often enabling small models to outperform larger ones, underscoring the practical impact of integrating external tools into sLM self-verification.

Abstract

Recent studies have demonstrated that test-time compute scaling effectively improves the performance of small language models (sLMs). However, prior research has mainly examined test-time compute scaling with an additional larger model as a verifier, leaving self-verification by sLMs underexplored. In this work, we investigate whether sLMs can reliably self-verify their outputs under test-time scaling. We find that even with knowledge distillation from larger verifiers, sLMs struggle with verification tasks requiring memorization, such as numerical calculations and fact-checking. To address this limitation, we propose Tool-integrated self-verification (T1), which delegates memorization-heavy verification steps to external tools, such as a code interpreter. Our theoretical analysis shows that tool integration reduces memorization demands and improves test-time scaling performance. Experiments on the MATH benchmark demonstrate that, with T1, a Llama-3.2 1B model under test-time scaling outperforms the significantly larger Llama-3.1 8B model. Moreover, T1 generalizes effectively to both mathematical (MATH500) and multi-domain knowledge-intensive tasks (MMLU-Pro). Our findings highlight the potential of tool integration to substantially improve the self-verification abilities of sLMs.

Figures (12)

• Figure 1: (a) Concept figure. Small language models (sLMs) often fail due to their limited capacity. However, when sLMs utilize external tools, their reliability significantly improves. (b) Concept-proof experimental results. We evaluate Llama 1B model on their ability to verify arithmetic calculations of $N$ 3-digit numbers. The performance of 1B model consistently drops as $N$ increases. However, enabling code generation and execution for verification largely mitigates the performance drop. See \ref{['appendix:concpet-proof']} for details of concept-proof experiments.
• Figure 2: Tool-integrated self-verification for mathematical reasoning. (a) Generator: A small language model (sLM) may produce incorrect solutions due to calculation errors. (b) Tool-based Verifier (ToolV): The sLM generates executable code based on its reasoning; the output of the code is used to verify the solution’s correctness. (c) Reward Model (RM)-based Verifier: The reward model (GenRM / PRM) still evaluates the solution as before, but its verdict only contributes to the final decision if the solution passes the tool-assisted filter. Concrete examples are in \ref{['appendix:sec:case_analysis']}.
• Figure 3: MATH500 with PRM. Weighted Best-of-N performance of three small language models, emphasizing the benefits of ToolV on college-level math problems. ToolV significantly enhances PRM, enabling small models to outperform or match much larger models. Qwen2.5-1.5B and Llama3.1-8B performances are reported as $N=1$ greedy decoding.
• Figure 4: MATH500 with GenRM. Weighted Best-of-N performance of three small language models, showcasing the effectiveness of ToolV with GenRM, where even generative verification cannot supplement the calculation error which can be easily filtered out by using a tool.
• Figure 5: GSM8K with GenRM. Weighted Best-of-N performance comparison across three small language models. The results show that ToolV also improves model performance on graduate-level arithmetic problems. However, the gains are smaller on this simpler task, where existing verifiers already perform reliably compared to more challenging tasks.

Theorems & Definitions (11)

• Lemma 5.1: Memorization without Tool MemorizationCapacity
• proof : Proof sketch
• Theorem 5.2: Memorization with Tool
• proof : Proof sketch
• Theorem 5.3: Monotonicity of Imperfect Verifier
• proof : Proof sketch
• proof
• proof
• Theorem C.1: Best-of-$N$ Accuracy with Imperfect Verifier
• proof