Will It Survive? Deciphering the Fate of AI-Generated Code in Open Source
Musfiqur Rahman, Emad Shihab
TL;DR
This study tackles the disposable-code hypothesis by applying survival analysis to track agent-authored versus human-authored code across 201 OSS projects in the AIDev dataset. Using line-level granularity, Kaplan-Meier, log-rank tests, and Cox regression, it finds that agent-authored code survives longer, with a line-level hazard ratio of $HR=0.842$ ($p<0.001$) indicating lower modification risk, though tool-specific effects vary (Copilot-style tools show stronger longevity, Devin less so). When modifications occur, agent-generated code shows more corrective and preventive changes, but with a small overall effect size. Predicting whether code will be modified from birth yields modest accuracy ($\text{AUC-ROC}=0.671$, $\text{Macro F1}=0.285$), while forecasting when modifications happen is substantially harder, dominated by file history and organizational factors rather than authorship. The findings argue that organizational practices and ownership, rather than generation quality alone, govern the long-term evolution of AI-generated code, with replication artifacts and actionable guidance for ownership, review, and tool selection provided for practitioners and researchers alike.
Abstract
The integration of AI agents as coding assistants into software development has raised questions about the long-term viability of AI agent-generated code. A prevailing hypothesis within the software engineering community suggests this code is "disposable", meaning it is merged quickly but discarded shortly thereafter. If true, organizations risk shifting maintenance burden from generation to post-deployment remediation. We investigate this hypothesis through survival analysis of 201 open-source projects, tracking over 200,000 code units authored by AI agents versus humans. Contrary to the disposable code narrative, agent-authored code survives significantly longer: at the line level, it exhibits a 15.8 percentage-point lower modification rate and 16% lower hazard of modification (HR = 0.842, p < 0.001). However, modification profiles differ. Agent-authored code shows modestly elevated corrective rates (26.3% vs. 23.0%), while human code shows higher adaptive rates. However, the effect sizes are small (Cramér's V = 0.116), and per-agent variation exceeds the agent-human gap. Turning to prediction, textual features can identify modification-prone code (AUC-ROC = 0.671), but predicting when modifications occur remains challenging (Macro F1 = 0.285), suggesting timing depends on external organizational dynamics. The bottleneck for agent-generated code may not be generation quality, but the organizational practices that govern its long-term evolution.