CodeI/O: Condensing Reasoning Patterns via Code Input-Output Prediction

TL;DR

CodeI/O addresses sparse supervision in many reasoning tasks by leveraging reasoning patterns embedded in real-world code. It transforms code into a code input-output prediction task with natural language CoTs, enabling the model to learn universal reasoning primitives. A large CodeI/O dataset and a revision-enhanced CodeI/O++ extend the training signals; two-stage training before instruction tuning yields consistent, balanced improvements across 14 benchmarks and multiple base models. The work demonstrates that code-based reasoning signals can generalize beyond programming tasks and provides insights into data design, synthesis models, and revision strategies.

Abstract

Reasoning is a fundamental capability of Large Language Models. While prior research predominantly focuses on enhancing narrow skills like math or code generation, improving performance on many other reasoning tasks remains challenging due to sparse and fragmented training data. To address this issue, we propose CodeI/O, a novel approach that systematically condenses diverse reasoning patterns inherently embedded in contextually-grounded codes, through transforming the original code into a code input-output prediction format. By training models to predict inputs/outputs given code and test cases entirely in natural language as Chain-of-Thought (CoT) rationales, we expose them to universal reasoning primitives -- like logic flow planning, state-space searching, decision tree traversal, and modular decomposition -- while decoupling structured reasoning from code-specific syntax and preserving procedural rigor. Experimental results demonstrate CodeI/O leads to consistent improvements across symbolic, scientific, logic, math & numerical, and commonsense reasoning tasks. By matching the existing ground-truth outputs or re-executing the code with predicted inputs, we can verify each prediction and further enhance the CoTs through multi-turn revision, resulting in CodeI/O++ and achieving higher performance. Our data and models are available at https://github.com/hkust-nlp/CodeIO.

Figures (8)

• Figure 1: Overview of our training data construction: Raw code files are gathered from various sources and converted into a unified format. Input-output pairs are then generated by executing the code, while natural language CoTs for predictions are collected from DeepSeek-V2.5. The verified CoTs can undergo optional revisions to further enhance reasoning chains.
• Figure 2: Two examples for the collected responses for input and output prediction respectively.
• Figure 3: Average scores of Stage 1 training on CodeI/O, a 3.5M WebInstruct subset (WI) and an enhanced version distilled from DeepSeek-V2.5 Directly (WI-DS25).
• Figure 4: The scaling effect of CodeI/O in the first stage training.
• Figure 5: Average benchmark scores from training on data from different turns of revision.