Teaching Language Models to Reason with Tools

TL;DR

This paper introduces CoRT, a post-training framework that teaches large reasoning models to effectively leverage external code interpreters for mathematical problem solving. It combines Hint-Engineering with a minimal, high-quality data budget (about 30 manual samples) and a three-stage training pipeline (Cold Start, SFT, RFT, plus RL for smaller models) to achieve significant accuracy gains and substantial token efficiency across challenging benchmarks. Key innovations include strategic hint insertion to guide CI usage, rejection fine-tuning to filter poor trajectories, and a strong-to-weak distillation path to smaller models, augmented by a tailored GRPO-based reinforcement learning regime with persistent execution and code-execution rewards. Empirical results show absolute accuracy increases (up to ~8%) and large token savings (roughly 30–50%), with strong generalization demonstrated in an out-of-distribution chemistry task and spontaneous DL tool discovery, underscoring the practical impact for scalable, efficient tool-augmented reasoning.

Abstract

Large reasoning models (LRMs) like OpenAI-o1 have shown impressive capabilities in natural language reasoning. However, these models frequently demonstrate inefficiencies or inaccuracies when tackling complex mathematical operations. While integrating computational tools such as Code Interpreters (CIs) offers a promising solution, it introduces a critical challenge: a conflict between the model's internal, probabilistic reasoning and the external, deterministic knowledge provided by the CI, which often leads models to unproductive deliberation. To overcome this, we introduce CoRT (Code-Optimized Reasoning Training), a post-training framework designed to teach LRMs to effectively utilize CIs. We propose \emph{Hint-Engineering}, a new data synthesis strategy that strategically injects diverse hints at optimal points within reasoning paths. This approach generates high-quality, code-integrated reasoning data specifically tailored to optimize LRM-CI interaction. Using this method, we have synthesized 30 high-quality samples to post-train models ranging from 1.5B to 32B parameters through supervised fine-tuning. CoRT further refines the multi-round interleaving of external CI usage and internal thinking by employing rejection sampling and reinforcement learning. Our experimental evaluations demonstrate CoRT's effectiveness, yielding absolute improvements of 4\% and 8\% on DeepSeek-R1-Distill-Qwen-32B and DeepSeek-R1-Distill-Qwen-1.5B, respectively, across five challenging mathematical reasoning datasets. Moreover, CoRT significantly enhances efficiency, reducing token usage by approximately 30\% for the 32B model and 50\% for the 1.5B model compared to pure natural language reasoning baselines. The models and code are available at: https://github.com/ChengpengLi1003/CoRT.

Figures (11)

• Figure 2: The training framework of CoRT.
• Figure 3: Comparison between prompt-hint and hint-engineering approaches using Problem 13 from AIME23 I as a case study (prompt prefix omitted for brevity).Both methods begin with a general hint (in green) after <think> to encourage code usage. While prompt-hint (a) allows natural interaction between R1 and the Code Interpreter (CI), leading to inefficient token usage, hint-engineering (b) introduces strategic hints at key decision points. Hint-1 is inserted when the model begins manual calculation of complex volumes (V1 and V2), redirecting to Python computation. Hint-2 is added to prevent unnecessary verification of Python calculations. Through these targeted interventions, hint-engineering achieves approximately 5000 tokens reduction while maintaining solution accuracy. More examples are provided in \ref{['casestudy']}.
• Figure 4: Token efficiency analysis on AIME24. (a): Token efficiency comparison showing Hint-Engineering-RFT-32B achieves comparable accuracy with significantly fewer tokens (40-50% token saving) compared to Prompt-Hint-SFT-32B. (b): Average token usage for correct and incorrect responses across different models, with Hint-Engineering models maintaining lower token consumption while achieving competitive performance.
• Figure 5: Analysis of Python code usage patterns. (a) Distribution of code usage types: Hint-Engineering shows a preference for calculation while Prompt-Hint favors verification tasks. (b) and (c) Function-specific distribution: Hint-Engineering demonstrates more balanced usage across different Python functions compared to Prompt-Hint.
• Figure 6: Ablation study on the impact of code execution reward during RL training on AIME24. Left: Performance of Prompt-Hint-1.5B-RL with and without code reward. Right: Performance of Hint-Engineering-1.5B-RL with and without code reward. Both approaches show consistent performance improvements when trained with the additional code execution reward.