Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

When Many-Shot Prompting Fails: An Empirical Study of LLM Code Translation

Amirkia Rafiei Oskooei, Kaan Baturalp Cosdan, Husamettin Isiktas, Mehmet S. Aktas

TL;DR

This paper investigates whether increasing in-context demonstrations improves LLM-based code translation. Through a large-scale study across 30 language pairs and 90k translations using long-context Gemini models on the CodeTransOcean benchmark, the authors reveal a many-shot paradox: functional correctness peaks with few-shot prompts (5-25 examples) and degrades with larger context, even as static similarity metrics may rise. They show a divergence between static and dynamic evaluations, observe error-type shifts toward functional and runtime failures in many-shot regimes, and highlight substantial cost increases with large prompts. Practically, the findings argue for carefully curated, high-quality few-shot exemplars and hint at the need for iterative prompting approaches to maintain semantic correctness while keeping costs down.

Abstract

Large Language Models (LLMs) with vast context windows offer new avenues for in-context learning (ICL), where providing many examples ("many-shot" prompting) is often assumed to enhance performance. We investigate this assumption for the complex task of code translation. Through a large-scale empirical study of over 90,000 translations, we systematically evaluate the impact of scaling in-context examples from zero-shot to many-shot configurations of up to 625 examples, with prompts spanning from approximately 100,000 to 800,000 tokens. Our findings reveal a "many-shot paradox": while static similarity metrics may modestly improve with more examples, functional correctness consistently peaks with few-shot prompting (5-25 examples). Providing substantially more examples often degrades this crucial functional performance. This study highlights that for code translation, the quality of a few well-chosen examples outweighs sheer quantity, challenging the universal efficacy of "more is better" for ICL and underscoring the task-dependent nature of optimal prompting strategies. Our results have significant implications for effectively leveraging LLMs in software engineering.

When Many-Shot Prompting Fails: An Empirical Study of LLM Code Translation

TL;DR

This paper investigates whether increasing in-context demonstrations improves LLM-based code translation. Through a large-scale study across 30 language pairs and 90k translations using long-context Gemini models on the CodeTransOcean benchmark, the authors reveal a many-shot paradox: functional correctness peaks with few-shot prompts (5-25 examples) and degrades with larger context, even as static similarity metrics may rise. They show a divergence between static and dynamic evaluations, observe error-type shifts toward functional and runtime failures in many-shot regimes, and highlight substantial cost increases with large prompts. Practically, the findings argue for carefully curated, high-quality few-shot exemplars and hint at the need for iterative prompting approaches to maintain semantic correctness while keeping costs down.

Abstract

Large Language Models (LLMs) with vast context windows offer new avenues for in-context learning (ICL), where providing many examples ("many-shot" prompting) is often assumed to enhance performance. We investigate this assumption for the complex task of code translation. Through a large-scale empirical study of over 90,000 translations, we systematically evaluate the impact of scaling in-context examples from zero-shot to many-shot configurations of up to 625 examples, with prompts spanning from approximately 100,000 to 800,000 tokens. Our findings reveal a "many-shot paradox": while static similarity metrics may modestly improve with more examples, functional correctness consistently peaks with few-shot prompting (5-25 examples). Providing substantially more examples often degrades this crucial functional performance. This study highlights that for code translation, the quality of a few well-chosen examples outweighs sheer quantity, challenging the universal efficacy of "more is better" for ICL and underscoring the task-dependent nature of optimal prompting strategies. Our results have significant implications for effectively leveraging LLMs in software engineering.
Paper Structure (17 sections, 1 equation, 4 figures, 4 tables)

This paper contains 17 sections, 1 equation, 4 figures, 4 tables.

Table of Contents

  1. Introduction
  2. Related Works
  3. LLMs for Code Translation
  4. Many-Shot In-Context Learning
  5. Positioning Our Work
  6. Methodology
  7. Models
  8. Dataset and Prompting Configurations
  9. Evaluation Framework
  10. Static Evaluation
  11. Dynamic Evaluation
  12. Results and Analysis
  13. The Many-Shot Paradox: Functional Correctness Peaks with Few Shots
  14. Error Distribution Analysis
  15. Influence of Language Characteristics
  16. ...and 2 more sections

Figures (4)

  • Figure 1: An overview of our experimental methodology.
  • Figure 2: Distribution of evaluation outcomes (aggregated across all language pairs) for each Gemini model variant. Moving from zero-shot to few-shot (5-25) reduces Compilation Errors and increases Successful Translations. In many-shot settings, however, successful translations decline while Functional and Runtime Errors become more prevalent.
  • Figure 3: Outcome distribution by category; syntactically similar pairs yield higher success rates.
  • Figure 4: Performance metrics broken down by language pair category. The non-monotonic trend in Pass@1 (rightmost plot) is observed across three categories, demonstrating the robustness of the "many-shot paradox."