Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

How to Design Translation Prompts for ChatGPT: An Empirical Study

Yuan Gao, Ruili Wang, Feng Hou

TL;DR

The paper investigates how to unleash ChatGPT’s translation capabilities through carefully crafted prompts that encode the translation task, domain context, and optional POS information, plus few-shot exemplars. It conducts extensive experiments across multilingual, multi-reference, and multi-domain translation tasks, comparing prompts against a TP3 baseline and commercial systems. Results show that domain-aware prompts (and, to a mixed extent, POS-augmented prompts) can yield BLEU gains and even surpass commercial systems in certain directions, though effects vary by language pair and evaluation setup. The work highlights the importance of multi-reference evaluation to capture ChatGPT’s translation diversity and identifies promising directions for robust prompt design while noting instabilities that warrant further study.

Abstract

The recently released ChatGPT has demonstrated surprising abilities in natural language understanding and natural language generation. Machine translation relies heavily on the abilities of language understanding and generation. Thus, in this paper, we explore how to assist machine translation with ChatGPT. We adopt several translation prompts on a wide range of translations. Our experimental results show that ChatGPT with designed translation prompts can achieve comparable or better performance over commercial translation systems for high-resource language translations. We further evaluate the translation quality using multiple references, and ChatGPT achieves superior performance compared to commercial systems. We also conduct experiments on domain-specific translations, the final results show that ChatGPT is able to comprehend the provided domain keyword and adjust accordingly to output proper translations. At last, we perform few-shot prompts that show consistent improvement across different base prompts. Our work provides empirical evidence that ChatGPT still has great potential in translations.

How to Design Translation Prompts for ChatGPT: An Empirical Study

TL;DR

The paper investigates how to unleash ChatGPT’s translation capabilities through carefully crafted prompts that encode the translation task, domain context, and optional POS information, plus few-shot exemplars. It conducts extensive experiments across multilingual, multi-reference, and multi-domain translation tasks, comparing prompts against a TP3 baseline and commercial systems. Results show that domain-aware prompts (and, to a mixed extent, POS-augmented prompts) can yield BLEU gains and even surpass commercial systems in certain directions, though effects vary by language pair and evaluation setup. The work highlights the importance of multi-reference evaluation to capture ChatGPT’s translation diversity and identifies promising directions for robust prompt design while noting instabilities that warrant further study.

Abstract

The recently released ChatGPT has demonstrated surprising abilities in natural language understanding and natural language generation. Machine translation relies heavily on the abilities of language understanding and generation. Thus, in this paper, we explore how to assist machine translation with ChatGPT. We adopt several translation prompts on a wide range of translations. Our experimental results show that ChatGPT with designed translation prompts can achieve comparable or better performance over commercial translation systems for high-resource language translations. We further evaluate the translation quality using multiple references, and ChatGPT achieves superior performance compared to commercial systems. We also conduct experiments on domain-specific translations, the final results show that ChatGPT is able to comprehend the provided domain keyword and adjust accordingly to output proper translations. At last, we perform few-shot prompts that show consistent improvement across different base prompts. Our work provides empirical evidence that ChatGPT still has great potential in translations.
Paper Structure (13 sections, 4 figures, 5 tables)

This paper contains 13 sections, 4 figures, 5 tables.

Table of Contents

  1. Introduction
  2. Background
  3. Experiments
  4. Prompt Design
  5. Experimental Setup
  6. Datasets
  7. Baselines and Evaluation Metrics
  8. Multilingual Translation
  9. POS tags for ChatGPT
  10. Translation Diversity
  11. Multi-domain Translation
  12. Few-shot Prompts
  13. Conclusion and Further work

Figures (4)

  • Figure 1: Best performing translations answered by ChatGPT.
  • Figure 2: The template of few-shot prompts. We take TP3 as an example.
  • Figure 3: Comparing BLEU scores of ChatGPT with 0, 1, 5-shot prompts on the Fr$\rightarrow$En translation.
  • Figure 4: Comparing BLEU scores of ChatGPT with 0, 1, 5-shot prompts on the Es$\rightarrow$En translation.