When Code Crosses Borders: A Security-Centric Study of LLM-based Code Translation
Hailong Chang, Guozhu Meng, Shuhui Xiao, Kai Chen, Kun Sun, Yilin Li
TL;DR
This work provides the first large-scale, security-centric evaluation of LLM-based code translation, revealing substantial vulnerability introduction and preservation across languages and CWE types. It assembles a 720-sample, multi-language dataset spanning four languages and nine CWEs, and benchmarks five recent LLMs through 6,000 translation tasks, using rigorous human expert evaluation complemented by automated tools and a reference LLM-as-a-judge. The authors introduce a five-type taxonomy of translation-induced vulnerabilities and demonstrate a Retrieval-Augmented Generation (RAG) mitigation that reduces vulnerability introduction by about 32.8% (with some trade-off in functional correctness). Together, these findings offer concrete guidance for developing secure code translation pipelines and highlight a practical path—knowledge-enhanced prompting via RAG—for mitigating security risks in AI-assisted software migration.
Abstract
Code translation is crucial for cross-language codebase migration, and large language models (LLMs) have emerged as a promising technique to automate this process. However, the security implications of using LLMs for code translation remain largely unexplored, as existing evaluations primarily focus on syntactic and functional correctness. To bridge this gap, we conduct a security-centric empirical study to investigate the risks of vulnerabilities being introduced or preserved during LLM-based translation. Our study involves a rigorous evaluation of five state-of-the-art LLMs on a curated dataset of 720 security-related code samples across four programming languages (Java, PHP, C, C++) and nine Common Weakness Enumeration (CWE) categories. The results reveal significant security degradation, with 28.6\% to 45\% of translations introducing new vulnerabilities. Web-related flaws, particularly in input validation, proved most challenging for LLMs. Furthermore, we identify and categorize the root causes of these vulnerable translations into a taxonomy of five major error types. Based on our findings, we develop and evaluate a Retrieval-Augmented Generation (RAG)-based mitigation strategy, which successfully reduces the vulnerability introduction rate by 32.8\%. Our study provides the first large-scale evidence of serious security risks in LLM-based code translation and demonstrates the potential of knowledge-enhanced prompting to mitigate them.