On Code-Induced Reasoning in LLMs

TL;DR

This work presents a data-centric framework to disentangle which Code properties drive reasoning improvements in LLMs. By constructing parallel natural language and code instruction datasets across ten languages and applying both rule-based and generative perturbations, the authors quantify how structural and semantic aspects influence NL, math, and code tasks after finetuning multiple model families. They find that code structure is particularly critical for mathematical and code reasoning, but high-level abstractions like pseudocode and flowcharts can match or exceed raw code under certain conditions, and models remain robust to corrupted signals by leveraging surface regularities. Language-specific patterns emerge, with Python favoring NL reasoning and lower-level languages such as Java and Rust benefiting math, offering practical guidance for assembling training data to enhance LLM reasoning capabilities.

Abstract

Code data has been shown to enhance the reasoning capabilities of large language models (LLMs), but it remains unclear which aspects of code are most responsible. We investigate this question with a systematic, data-centric framework. We construct parallel instruction datasets in ten programming languages and apply controlled perturbations that selectively disrupt structural or semantic properties of code. We then finetune LLMs from five model families and eight scales on each variant and evaluate their performance on natural language, math, and code tasks. Across 3,331 experiments, our results show that LLMs are more vulnerable to structural perturbations than semantic ones, particularly on math and code tasks. Appropriate abstractions like pseudocode and flowcharts can be as effective as code, while encoding the same information with fewer tokens without adhering to original syntax can often retain or even improve performance. Remarkably, even corrupted code with misleading signals remains competitive when surface-level regularities persist. Finally, syntactic styles also shape task-specific gains with Python favoring natural language reasoning and lower-level languages such as Java and Rust favoring math. Through our systematic framework, we aim to provide insight into how different properties of code influence reasoning and inform the design of training data for enhancing LLM reasoning capabilities.

Figures (51)

• Figure 1: We construct parallel code and natural language instruction datasets, apply targeted modifications (rule-based and generative-based perturbations, single programming language ablations), and fine-tune a separate LLM on each modified dataset. We then evaluate the resulting models across general natural language, code, and math reasoning tasks.
• Figure 2: Performance (with stderr bars) of Qwen3-4B-Base across zero-shot, full code finetuning (code-ft), full natural language finetuning (nl-ft), and 50-50 code to NL data ratio finetuning (mixed ft). Incorporating code improves performance across tasks.
• Figure 3: Aggregated performance (with stderr bars) under structural perturbations (e.g. removing whitespace) vs. semantics perturbations (e.g. modifying the comments) of Qwen3-4B-Base. Semantic perturbations tend to be more harmful to performance than semantic ones.
• Figure 4: Aggregated performance (with stderr bars) under levels of explicitness of code structure (less explicit going from runnable code to NL procedure) of Qwen3-8B-Base. Certain algorithmic and graphical abstractions benefit reasoning.
• Figure 5: Aggregated performance (with stderr bars) of Qwen3-0.6B-Base with various of token counts wrt to unperturbed code. Reductions can perform comparable or even better than the baseline.