CoRT: Code-integrated Reasoning within Thinking

TL;DR

CoRT tackles the challenge of effectively integrating code interpreters into large reasoning models for complex mathematical tasks. It introduces Hint-Engineering to create high-quality, selectively annotated data and pairs it with a post-training pipeline (SFT/RFT and RL) to teach models when and how to use code tools. The approach yields consistent improvements across 32B and 1.5B models, notably in token efficiency and sample efficiency, and demonstrates how RL rewards can steer code usage toward more productive patterns. These results highlight a scalable path to more accurate and resource-efficient code-integrated reasoning in LRMs, with broad implications for math-heavy AI applications.

Abstract

Large Reasoning Models (LRMs) like o1 and DeepSeek-R1 have shown remarkable progress in natural language reasoning with long chain-of-thought (CoT), yet they remain inefficient or inaccurate when handling complex mathematical operations. Addressing these limitations through computational tools (e.g., computation libraries and symbolic solvers) is promising, but it introduces a technical challenge: Code Interpreter (CI) brings external knowledge beyond the model's internal text representations, thus the direct combination is not efficient. This paper introduces CoRT, a post-training framework for teaching LRMs to leverage CI effectively and efficiently. As a first step, we address the data scarcity issue by synthesizing code-integrated reasoning data through Hint-Engineering, which strategically inserts different hints at appropriate positions to optimize LRM-CI interaction. We manually create 30 high-quality samples, upon which we post-train models ranging from 1.5B to 32B parameters, with supervised fine-tuning, rejection fine-tuning and reinforcement learning. Our experimental results demonstrate that Hint-Engineering models achieve 4\% and 8\% absolute improvements on DeepSeek-R1-Distill-Qwen-32B and DeepSeek-R1-Distill-Qwen-1.5B respectively, across five challenging mathematical reasoning datasets. Furthermore, Hint-Engineering models use about 30\% fewer tokens for the 32B model and 50\% fewer tokens for the 1.5B model compared with the natural language models. The models and code are available at https://github.com/ChengpengLi1003/CoRT.

Figures (11)

• Figure 2: The training framework of CoRT.
• Figure 3: Comparison between prompt-hint and hint-engineering approaches using Problem 13 from AIME23 I as a case study (prompt prefix omitted for brevity).Both methods begin with a general hint (in green) after <think> to encourage code usage. While prompt-hint (a) allows natural interaction between R1 and the Code Interpreter (CI), leading to inefficient token usage, hint-engineering (b) introduces strategic hints at key decision points. Hint-1 is inserted when the model begins manual calculation of complex volumes (V1 and V2), redirecting to Python computation. Hint-2 is added to prevent unnecessary verification of Python calculations. Through these targeted interventions, hint-engineering achieves approximately 5000 token reduction while maintaining solution accuracy. More examples are provided in \ref{['casestudy']}.
• Figure 4: Token efficiency analysis on AIME24. (a): Token efficiency comparison showing Hint-Engineering-RFT-32B achieves comparable accuracy with significantly fewer tokens (40-50% token saving) compared to Prompt-Hint-SFT-32B. (b): Average token usage for correct and incorrect responses across different models, with Hint-Engineering models maintaining lower token consumption while achieving competitive performance.
• Figure 5: Analysis of Python code usage patterns. (a) Distribution of code usage types: Hint-Engineering shows a preference for calculation while Prompt-Hint favors verification tasks. (b) and (c) Function-specific distribution: Hint-Engineering demonstrates more balanced usage across different Python functions compared to Prompt-Hint.
• Figure 6: Ablation study on the impact of code execution reward during RL training on AIME24. Left: Performance of Prompt-Hint-1.5B-RL with and without code reward. Right: Performance of Hint-Engineering-1.5B-RL with and without code reward. Both approaches show consistent performance improvements when trained with the additional code execution reward.