TRACE: Evaluating Execution Efficiency of LLM-Based Code Translation

Abstract

While Large Language Models (LLMs) have substantially improved the functional correctness of code translation, the critical dimension of \textit{execution efficiency} remains overlooked. We present \textbf{\textsc{trace}}, the first benchmark to explicitly assess efficiency in LLM-translated code. \textsc{trace} includes 1,000 efficiency-critical tasks across C++, Java, and Python, each augmented with stress tests that reveal efficiency degradations often overlooked by small-scale tests. Using \textsc{trace}, we conduct an extensive evaluation of 28 representative LLMs and highlight several key insights: 1) Correctness is not a reliable proxy for efficiency: the correctness leader \textit{Claude-4-think} achieves only mid-level time efficiency, outperformed by smaller open-source LLMs such as \textit{Qwen2.5-Coder-14B-Instruct}. 2) Inefficiency is both prevalent and patterned: 23.5\% of correct translations exhibit pronounced inefficiency, distributed across algorithmic faults (11.9\%), language construct mismatches (66.4\%), and resource mismanagement (21.7\%). 3) Inference-time prompt strategies bring only modest improvements, suggesting that current LLMs lack intrinsic efficiency awareness. Together, our results establish efficiency as an essential dimension of code translation and position \textsc{trace} as a principled foundation for efficiency-oriented evaluation.

Figures (15)

• Figure 1: A motivating example illustrating the critical discrepancy between functional correctness and execution efficiency in LLM-based code translation.
• Figure 2: An overview of Trace's LLM-driven two-stage construction pipeline.
• Figure 3: Execution profile distribution of translations.
• Figure 4: OLS regression of correctness-conditioned Beyond score (efficiency) on Pass rate (correctness).
• Figure 5: A case study of Algorithm Implementation Discrepancy. The inefficient translation mistakenly converts an early-exit loop into one that iterates over the entire potential range, which degrades the original algorithm's time complexity and causes a catastrophic time slowdown of over 6400$\times$.