Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Not All Code Is Equal: A Data-Centric Study of Code Complexity and LLM Reasoning

Lukas Twist, Shu Yang, Hanqi Yan, Jingzhi Gong, Di Wang, Helen Yannakoudakis, Jie M. Zhang

TL;DR

This study probes whether the structural properties of code used during fine-tuning influence downstream reasoning in open-weight LLMs. By constructing two complexity-controlled datasets (solution-driven CodeNet and problem-driven Instruct) and evaluating a range of models on six reasoning benchmarks, the authors show that code-aware gains are not uniform and tend to peak at intermediate levels of structural complexity. Cyclomatic complexity provides a more stable and interpretable signal than logical lines of code, and absolute complexity is more predictive than how that complexity is produced. High complexity can even harm performance, and restricting training data to a specific complexity range often outperforms training on mixed-code corpora, suggesting a data-centric path to improving reasoning beyond scaling. The work highlights the value of targeted code-structure signals for reasoning and calls for richer, more nuanced metrics to guide data curation for LLM reasoning enhancement.

Abstract

Large Language Models (LLMs) increasingly exhibit strong reasoning abilities, often attributed to their capacity to generate chain-of-thought-style intermediate reasoning. Recent work suggests that exposure to code can further enhance these skills, but existing studies largely treat code as a generic training signal, leaving open the question of which properties of code actually contribute to improved reasoning. To address this gap, we study the structural complexity of code, which captures control flow and compositional structure that may shape how models internalise multi-step reasoning during fine-tuning. We examine two complementary settings: solution-driven complexity, where complexity varies across multiple solutions to the same problem, and problem-driven complexity, where complexity reflects variation in the underlying tasks. Using cyclomatic complexity and logical lines of code to construct controlled fine-tuning datasets, we evaluate a range of open-weight LLMs on diverse reasoning benchmarks. Our findings show that although code can improve reasoning, structural properties strongly determine its usefulness. In 83% of experiments, restricting fine-tuning data to a specific structural complexity range outperforms training on structurally diverse code, pointing to a data-centric path for improving reasoning beyond scaling.

Not All Code Is Equal: A Data-Centric Study of Code Complexity and LLM Reasoning

TL;DR

This study probes whether the structural properties of code used during fine-tuning influence downstream reasoning in open-weight LLMs. By constructing two complexity-controlled datasets (solution-driven CodeNet and problem-driven Instruct) and evaluating a range of models on six reasoning benchmarks, the authors show that code-aware gains are not uniform and tend to peak at intermediate levels of structural complexity. Cyclomatic complexity provides a more stable and interpretable signal than logical lines of code, and absolute complexity is more predictive than how that complexity is produced. High complexity can even harm performance, and restricting training data to a specific complexity range often outperforms training on mixed-code corpora, suggesting a data-centric path to improving reasoning beyond scaling. The work highlights the value of targeted code-structure signals for reasoning and calls for richer, more nuanced metrics to guide data curation for LLM reasoning enhancement.

Abstract

Large Language Models (LLMs) increasingly exhibit strong reasoning abilities, often attributed to their capacity to generate chain-of-thought-style intermediate reasoning. Recent work suggests that exposure to code can further enhance these skills, but existing studies largely treat code as a generic training signal, leaving open the question of which properties of code actually contribute to improved reasoning. To address this gap, we study the structural complexity of code, which captures control flow and compositional structure that may shape how models internalise multi-step reasoning during fine-tuning. We examine two complementary settings: solution-driven complexity, where complexity varies across multiple solutions to the same problem, and problem-driven complexity, where complexity reflects variation in the underlying tasks. Using cyclomatic complexity and logical lines of code to construct controlled fine-tuning datasets, we evaluate a range of open-weight LLMs on diverse reasoning benchmarks. Our findings show that although code can improve reasoning, structural properties strongly determine its usefulness. In 83% of experiments, restricting fine-tuning data to a specific structural complexity range outperforms training on structurally diverse code, pointing to a data-centric path for improving reasoning beyond scaling.
Paper Structure (51 sections, 5 figures, 3 tables)

This paper contains 51 sections, 5 figures, 3 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Reasoning abilities of LLMs.
  4. Using code to enhance LLM reasoning.
  5. Code complexity metrics.
  6. Methodology
  7. Problem Formulation
  8. Code Complexity Metrics
  9. Metric selection.
  10. Metric calculation.
  11. Dataset Construction
  12. Solution-driven complexity.
  13. Problem-driven complexity.
  14. Natural language baseline.
  15. Evaluation Set-up
  16. ...and 36 more sections

Figures (5)

  • Figure 1: Overview of our experimental pipeline. We construct two complementary datasets that control structural code complexity through solution-driven complexity (top) and problem-driven complexity (bottom), creating twelve splits for each (five different complexity levels and one control, for two different complexity metrics). We then use these splits to fine-tune six models from three model families and evaluate downstream reasoning performance across six widely-used reasoning benchmarks.
  • Figure 2: Comparison of the code metrics used in this study. We calculate both metrics for our solution-driven complexity dataset (CodeNet) and our problem-driven complexity dataset (Instruct). Together they allow us to disentangle the effects of structural complexity from size-based complexity.
  • Figure 3: Per-model reasoning performance across complexity splits. Reasoning accuracy for each model after fine-tuning on complexity-controlled dataset splits. Solid lines correspond to solution-driven complexity splits (CodeNet), dashed lines to problem-driven complexity splits (Instruct); dotted horizontal lines indicate the corresponding results for control (ctrl) datasets with mixed complexity; the dash-dotted green line denotes the model's natural language (NL) baseline after fine-tuning on a strictly non-code dataset.
  • Figure 4: Average reasoning change vs. code complexity. Average accuracy change compared to the NL baseline across all six reasoning benchmarks as a function of cyclomatic complexity (CC, left) and logical lines of code (LLOC, right). Results are shown as a single line that includes both the solution-driven (CodeNet; circles) and the problem-driven (Instruct; squares) complexity datasets.
  • Figure 5: Correlation between code complexity and downstream reasoning accuracy. Spearman correlations between benchmark accuracy and training-data complexity level, computed across complexity-controlled dataset splits. Results are shown separately for solution-driven (CodeNet) and problem-driven (Instruct) settings, using cyclomatic complexity (CC) and logical lines of code (LLOC) as structural measures. Correlation calculation is detailed in Appendix \ref{['app:correlation']}.