Guided Debugging of Auto-Translated Code Using Differential Testing

TL;DR

tHinter addresses translation errors in auto-translated code by framing debugging as a differential-testing problem between source and translated programs. It combines coverage-guided fuzzing to generate diverse test cases with a two-layer localization algorithm that merges statistics and developer-experience heuristics to pinpoint problematic lines. In experiments on Python-to-C++ translations of LeetCode solutions, tHinter reduced manual debugging effort by $71\%$ and increased the likelihood that an LLM fixes errors in a single query by $59\%$, while also improving perceived fix quality. These results demonstrate that automatic translation-error localization can meaningfully augment both human and LLM-driven debugging workflows and support open-source dissemination.

Abstract

Large Language Models (LLMs) hold great promise in the task of code translation. However, the lack of explainability complicates the identification of the inevitable translation errors. In this paper, we propose tHinter, a debugging tool to locate translation errors in auto-translated code. The core idea of tHinter is that correctly translated, the source and translated code should present the same functionalities, giving the same output for the same input. Hence, lines in the translated code responsible for output differences are possibly translation errors. First, tHinter employs fuzzing to generate diverse test cases that thoroughly explore the translated code. Then, tHinter relies on a heuristic algorithm to pinpoint translation errors from coverage information and differential testing execution results of those test cases. This heuristic algorithm is designed to leverage both the statistics and the expertise of developers. Comprehensive experiments with real code show its effectiveness. It reduces 71% lines developers need to review during debugging and increases the likelihood of the LLM fixing translation errors in a single query by 59%. Developers generally consider it satisfactory and helpful.

Figures (7)

• Figure 2: Overview of tHinter
• Figure 3: A simplified demonstration of the prompt used in the translation step.
• Figure 4: A simplified demonstration of the prompt used in the validation step.
• Figure 5: tHinter's Performance Evaluated by $R_{reduc}$.
• Figure 6: Fixing translation errors by the LLM with/without tHinter.