Revisiting the Role of Natural Language Code Comments in Code Translation

TL;DR

This study systematically investigates whether natural language code comments improve LLM-based code translation across five programming languages using over 1100 samples and more than 80K translations from benchmarks like AVATAR and CodeNet. Through a large-scale empirical analysis, it explores how comment usefulness, intent, density, language, and placement affect translation quality, revealing that comments can both help and hinder depending on configuration. To harness benefits while mitigating downsides, the authors propose COMMENTRA, an iterative translation framework that translates uncommented code first and only injects comments for translations that fail, achieving substantial gains (up to 71% after iterative steps) across models and benchmarks. The work highlights practical guidance for benchmark design and real-world translation pipelines, showing that targeted, minimal commentary can significantly boost LLM-based code translation performance and potentially double it in some setups.

Abstract

The advent of large language models (LLMs) has ushered in a new era in automated code translation across programming languages. Since most code-specific LLMs are pretrained on well-commented code from large repositories like GitHub, it is reasonable to hypothesize that natural language code comments could aid in improving translation quality. Despite their potential relevance, comments are largely absent from existing code translation benchmarks, rendering their impact on translation quality inadequately characterised. In this paper, we present a large-scale empirical study evaluating the impact of comments on translation performance. Our analysis involves more than $80,000$ translations, with and without comments, of $1100+$ code samples from two distinct benchmarks covering pairwise translations between five different programming languages: C, C++, Go, Java, and Python. Our results provide strong evidence that code comments, particularly those that describe the overall purpose of the code rather than line-by-line functionality, significantly enhance translation accuracy. Based on these findings, we propose COMMENTRA, a code translation approach, and demonstrate that it can potentially double the performance of LLM-based code translation. To the best of our knowledge, our study is the first in terms of its comprehensiveness, scale, and language coverage on how to improve code translation accuracy using code comments.

Figures (19)

• Figure 1: Experimental Setup; The exact prompts used are also shown here.
• Figure 2: Venn diagrams depicting increase and decrease in LLMs performance in the commented code samples. Left and center diagrams show the overlap between uncommented successful and successfully translated model-commented samples; the right diagrams show the overlap between the various successfully translated model-commented samples.
• Figure 3: Bird's eye view of how various translation models performed on commented and uncommented code.
• Figure 4: Commenting Model Comparison. Size of the green bubbles $\propto$ no. of (source-PL, target-PL) pairs for which the corresponding (comment-model, translation-model) pair showed positive gains over the baseline.
• Figure 5: Example of one big multi-line (12 lines) comment with several intents