Correctness isnt Efficiency: Runtime Memory Divergence in LLM-Generated Code

TL;DR

The paper addresses the gap between functional correctness and reliable runtime behavior in LLM-generated code by introducing a stability-focused framework. It deploys Monotonic Peak Profiles ($MPP$) and Dynamic Time Warping ($DTW$) to derive Dynamic Mean Pairwise Distance ($DMPD$) and Model Instability Score ($MIS$), plus Normalized Maximum Velocity ($NMV$ to capture burstiness. Across CodeContests and BigOBench, it shows substantial memory-divergence among correct solutions, with higher sampling temperature increasing instability even as pass@1 may improve. The work also links stability metrics to software-engineering indicators, suggesting stability-aware reranking in CI/CD to reduce operational risk without sacrificing correctness. Practically, it provides a method to quantify and manage the runtime risk of AI-generated patches, with artifacts available for replication and extension to industry-scale workflows.

Abstract

Large language models (LLMs) can generate programs that pass unit tests, but passing tests does not guarantee reliable runtime behavior. We find that different correct solutions to the same task can show very different memory and performance patterns, which can lead to hidden operational risks. We present a framework to measure execution-time memory stability across multiple correct generations. At the solution level, we introduce Dynamic Mean Pairwise Distance (DMPD), which uses Dynamic Time Warping to compare the shapes of memory-usage traces after converting them into Monotonic Peak Profiles (MPPs) to reduce transient noise. Aggregating DMPD across tasks yields a model-level Model Instability Score (MIS). Experiments on BigOBench and CodeContests show substantial runtime divergence among correct solutions. Instability often increases with higher sampling temperature even when pass@1 improves. We also observe correlations between our stability measures and software engineering indicators such as cognitive and cyclomatic complexity, suggesting links between operational behavior and maintainability. Our results support stability-aware selection among passing candidates in CI/CD to reduce operational risk without sacrificing correctness. Artifacts are available.

Figures (11)

• Figure 1: Monotonic Peak Profiles (MPP) on BigOBench illustrating two regimes of runtime memory behavior across correct generations: left—divergence where one MPP differs significantly (high pairwise DMPD); right—consistency where profiles remain closely aligned (low pairwise DMPD).
• Figure 2: Example: DTW sequence alignment
• Figure 3: Example of correct solutions with runtime divergence because of different space algorithmic strategies. Solution A on the left uses ($O(nk)$ space while Solution B on the right uses $O(k)$.
• Figure 4: Pipeline for code generation and memory profiling
• Figure 5: MIS macro for BigObench and CodeContests