Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Understanding by Reconstruction: Reversing the Software Development Process for LLM Pretraining

Zhiyuan Zeng, Yichi Zhang, Yong Shan, Kai Hua, Siyuan Fang, Zhaiyu Liu, Jiaheng Liu, Haozhe Wang, Yining Zheng, Ming Ding, Ke Shen, Ge Zhang, Wenhao Huang, Xipeng Qiu

Abstract

While Large Language Models (LLMs) have achieved remarkable success in code generation, they often struggle with the deep, long-horizon reasoning required for complex software engineering. We attribute this limitation to the nature of standard pre-training data: static software repositories represent only the terminal state of an intricate intellectual process, abstracting away the intermediate planning, debugging, and iterative refinement. To bridge this gap, we propose a novel paradigm: understanding via reconstruction. We hypothesize that reverse-engineering the latent agentic trajectories -- the planning, reasoning, and debugging steps -- behind static repositories provides a far richer supervision signal than raw code alone. To operationalize this, we introduce a framework that synthesizes these trajectories using a multi-agent simulation. This process is grounded in the structural realities of the source repositories (e.g., dependency graphs and file hierarchies) to ensure fidelity. Furthermore, to guarantee the logical rigor of the synthetic data, we employ a search-based optimization technique that iteratively refines the Chain-of-Thought (CoT) reasoning to maximize the likelihood of the ground-truth code. Empirical results demonstrate that continuous pre-training on these reconstructed trajectories significantly enhances Llama-3-8B's performance across diverse benchmarks, including long-context understanding, coding proficiency, and agentic capabilities.

Understanding by Reconstruction: Reversing the Software Development Process for LLM Pretraining

Abstract

While Large Language Models (LLMs) have achieved remarkable success in code generation, they often struggle with the deep, long-horizon reasoning required for complex software engineering. We attribute this limitation to the nature of standard pre-training data: static software repositories represent only the terminal state of an intricate intellectual process, abstracting away the intermediate planning, debugging, and iterative refinement. To bridge this gap, we propose a novel paradigm: understanding via reconstruction. We hypothesize that reverse-engineering the latent agentic trajectories -- the planning, reasoning, and debugging steps -- behind static repositories provides a far richer supervision signal than raw code alone. To operationalize this, we introduce a framework that synthesizes these trajectories using a multi-agent simulation. This process is grounded in the structural realities of the source repositories (e.g., dependency graphs and file hierarchies) to ensure fidelity. Furthermore, to guarantee the logical rigor of the synthetic data, we employ a search-based optimization technique that iteratively refines the Chain-of-Thought (CoT) reasoning to maximize the likelihood of the ground-truth code. Empirical results demonstrate that continuous pre-training on these reconstructed trajectories significantly enhances Llama-3-8B's performance across diverse benchmarks, including long-context understanding, coding proficiency, and agentic capabilities.
Paper Structure (43 sections, 1 equation, 3 figures, 6 tables)

This paper contains 43 sections, 1 equation, 3 figures, 6 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Reverse Reasoning in Pretraining Data
  4. Synthetic Agent Trajectories
  5. Synthetic Data for Coding
  6. Approach
  7. Multi-Agent Trajectory Curation
  8. Main Agent: Project Planning
  9. Sub-Agent: File Implementation
  10. Grounding the Simulation with Extracted Information
  11. CoT Optimization via Search
  12. Continue Pretraining on Synthetic Agent Trajectories
  13. Trajectory Flattening
  14. Targeted Loss Masking
  15. Experiments
  16. ...and 28 more sections

Figures (3)

  • Figure 1: The pipeline of synthetic agent trajectory curation.
  • Figure 2: (a): Token distribution on thinking, too-call and too-response of main-agent and sub-agent. (b): Average number of tokens for each repo.
  • Figure 3: (a): The CoT increases with more CoT-optimization iterations. (b): PPL of the code to be generated decreases with more iterations.