Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

TransLibEval: Demystify Large Language Models' Capability in Third-party Library-targeted Code Translation

Pengyu Xue, Kunwu Zheng, Zhen Yang, Yifei Pei, Linhao Wu, Jiahui Dong, Xiapu Luo, Yan Xiao, Fei Liu, Yuxuan Zhang, Xiran Lyu, Xianhang Li, Xuanyu Zhu, Chengyi Wang

TL;DR

This work introduces TransLibEval, the first benchmark focused on library-centric code translation to rigorously evaluate LLM performance when Third-Party Libraries (TPLs) are involved. By providing 200 parallel tasks across Python, Java, and C++ with extensive library coverage and six translation strategies (Direct, IR-guided variants, and Retrieval-Augmented variants), the study reveals a substantial performance gap relative to library-free benchmarks and highlights the critical role of API/library awareness. A large-scale, expert-driven analysis of 4,831 GPT-4o failures uncovers a dominance of third-party reference errors and reveals strategy-specific failure modes, offering practical guidance for building more TPL-aware code intelligence. The results show that commercial LLMs generally outperform smaller models, Python translations are comparatively easier, and retrieval or IR-guided approaches can mitigate some library-related challenges, informing future directions for robust cross-language API mapping and library adaptation.

Abstract

In recent years, Large Language Models (LLMs) have been widely studied in the code translation field on the method, class, and even repository levels. However, most of these benchmarks are limited in terms of Third-Party Library (TPL) categories and scales, making TPL-related errors hard to expose and hindering the development of targeted solutions. Considering the high dependence (over 90%) on TPLs in practical programming, demystifying and analyzing LLMs' code translation performance involving various TPLs becomes imperative. To address this gap, we construct TransLibEval, the first benchmark dedicated to library-centric code translation. It consists of 200 real-world tasks across Python, Java, and C++, each explicitly involving TPLs from diverse categories such as data processing, machine learning, and web development, with comprehensive dependency coverage and high-coverage test suites. We evaluate seven recent LLMs of commercial, general, and code-specialized families under six translation strategies of three categories: Direct, IR-guided, and Retrieval-augmented. Experimental results show a dramatic performance drop compared with library-free settings (average CA decline over 60%), while diverse strategies demonstrate heterogeneous advantages. Furthermore, we analyze 4,831 failed cases from GPT-4o, one of the State-of-the-Art (SOTA) LLMs, revealing numerous third-party reference errors that were obscured previously. These findings highlight the unique challenges of library-centric translation and provide practical guidance for improving TPL-aware code intelligence.

TransLibEval: Demystify Large Language Models' Capability in Third-party Library-targeted Code Translation

TL;DR

This work introduces TransLibEval, the first benchmark focused on library-centric code translation to rigorously evaluate LLM performance when Third-Party Libraries (TPLs) are involved. By providing 200 parallel tasks across Python, Java, and C++ with extensive library coverage and six translation strategies (Direct, IR-guided variants, and Retrieval-Augmented variants), the study reveals a substantial performance gap relative to library-free benchmarks and highlights the critical role of API/library awareness. A large-scale, expert-driven analysis of 4,831 GPT-4o failures uncovers a dominance of third-party reference errors and reveals strategy-specific failure modes, offering practical guidance for building more TPL-aware code intelligence. The results show that commercial LLMs generally outperform smaller models, Python translations are comparatively easier, and retrieval or IR-guided approaches can mitigate some library-related challenges, informing future directions for robust cross-language API mapping and library adaptation.

Abstract

In recent years, Large Language Models (LLMs) have been widely studied in the code translation field on the method, class, and even repository levels. However, most of these benchmarks are limited in terms of Third-Party Library (TPL) categories and scales, making TPL-related errors hard to expose and hindering the development of targeted solutions. Considering the high dependence (over 90%) on TPLs in practical programming, demystifying and analyzing LLMs' code translation performance involving various TPLs becomes imperative. To address this gap, we construct TransLibEval, the first benchmark dedicated to library-centric code translation. It consists of 200 real-world tasks across Python, Java, and C++, each explicitly involving TPLs from diverse categories such as data processing, machine learning, and web development, with comprehensive dependency coverage and high-coverage test suites. We evaluate seven recent LLMs of commercial, general, and code-specialized families under six translation strategies of three categories: Direct, IR-guided, and Retrieval-augmented. Experimental results show a dramatic performance drop compared with library-free settings (average CA decline over 60%), while diverse strategies demonstrate heterogeneous advantages. Furthermore, we analyze 4,831 failed cases from GPT-4o, one of the State-of-the-Art (SOTA) LLMs, revealing numerous third-party reference errors that were obscured previously. These findings highlight the unique challenges of library-centric translation and provide practical guidance for improving TPL-aware code intelligence.

Paper Structure

This paper contains 21 sections, 4 equations, 5 figures, 6 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Large Language Models for Code Translation
  4. Existing Benchmarks for Code Translation
  5. New Benchmark: TransLibEval
  6. Calibration Construction
  7. Counterpart Construction
  8. Experimental Design
  9. Studied LLMs
  10. Studied Translation Strategies
  11. Evaluation Metrics
  12. Implementation Details
  13. Experimental Results
  14. RQ1: Overall Correctness
  15. RQ2: Translation Strategies
  16. ...and 6 more sections

Figures (5)

  • Figure 1: Performance Comparison between TransLibEval and Yang et al. among Diverse LLMs
  • Figure 2: Effectiveness of the RA(Name) Strategy Across Different Target Languages. (a) Successful retrieval of relevant C++ code patterns for CalculateDistance(). (b) Unsuccessful retrieval in Java, returning irrelevant Android-specific code.
  • Figure 3: Performance of Different Translation Strategies on TransLibEval
  • Figure 4: The Library-Dependency Awareness Analysis across Different LLMs and Translation Directions
  • Figure 5: A Horizontal Comparison of Error Analysis among Different Translation Strategies