Automated Multi-Language to English Machine Translation Using Generative Pre-Trained Transformers

TL;DR

This paper addresses offline multilingual translation into English using 16 open-source GPT models without fine-tuning. It benchmarks sentence-wise, zero-shot translations against a TED Talk-derived dataset and compares results with the Google Translate API, while logging wall-clock times on a single Nvidia A100 GPU. The key findings show mean translation quality across languages as BLEU $0.152$, GLEU $0.256$, chrF $0.448$, and METEOR $0.438$, with the best models being ReMM-v2-L2-13B for BLEU and METEOR and Llama2-chat-AYT-13B for chrF. The results reveal substantial language-specific variation, with French and Chinese occasionally surpassing Google Translate on some metrics, while several low-resource languages (e.g., Mongolian, Kazakh, Burmese) remain challenging, and the overall takeaway emphasizes the privacy and deployment advantages of offline, local inference despite generally lower accuracy.

Abstract

The task of accurate and efficient language translation is an extremely important information processing task. Machine learning enabled and automated translation that is accurate and fast is often a large topic of interest in the machine learning and data science communities. In this study, we examine using local Generative Pretrained Transformer (GPT) models to perform automated zero shot black-box, sentence wise, multi-natural-language translation into English text. We benchmark 16 different open-source GPT models, with no custom fine-tuning, from the Huggingface LLM repository for translating 50 different non-English languages into English using translated TED Talk transcripts as the reference dataset. These GPT model inference calls are performed strictly locally, on single A100 Nvidia GPUs. Benchmark metrics that are reported are language translation accuracy, using BLEU, GLEU, METEOR, and chrF text overlap measures, and wall-clock time for each sentence translation. The best overall performing GPT model for translating into English text for the BLEU metric is ReMM-v2-L2-13B with a mean score across all tested languages of $0.152$, for the GLEU metric is ReMM-v2-L2-13B with a mean score across all tested languages of $0.256$, for the chrF metric is Llama2-chat-AYT-13B with a mean score across all tested languages of $0.448$, and for the METEOR metric is ReMM-v2-L2-13B with a mean score across all tested languages of $0.438$.

Figures (50)

• Figure 1: Spanish-to-English dataset per-sentence translation quality and timing statistics for each of the $16$ GPT models. Timing is reported in the top sub-plot, on a log scale y-axis, using direct wall-clock compute time to produce the generated text per sentence. Datapoints which are smaller in the time plot mean that the GTP model output took less wall-clock time to generate. The bottom four sub-plots report the distribution of language quality metrics (one datapoint for each sentence), using the four different language quality measures. For all four of the language quality translation plots, scores closer to $1$ indicate better translation quality, and scores near $0$ indicate bad translation quality. All distributions are shown as box-plot representations, where the red dots indicate outlier points and the blue rectangles indicate the region between the first and third quartile's, the orange line denotes the median.
• Figure 2: French-to-English dataset per-sentence translation quality and timing statistics for each of the $16$ GPT models. Timing is reported in the top sub-plot, on a log scale y-axis, using direct wall-clock compute time to produce the generated text per sentence. Datapoints which are smaller in the time plot mean that the GTP model output took less wall-clock time to generate. The bottom four sub-plots report the distribution of language quality metrics (one datapoint for each sentence), using the four different language quality measures. For all four of the language quality translation plots, scores closer to $1$ indicate better translation quality, and scores near $0$ indicate bad translation quality. All distributions are shown as box-plot representations, where the red dots indicate outlier points and the blue rectangles indicate the region between the first and third quartile's, the orange line denotes the median.
• Figure 3: Chinese-to-English dataset per-sentence translation quality and timing statistics
• Figure 4: Arabic-to-English dataset per-sentence translation quality and timing statistics
• Figure 5: Hindi-to-English dataset per-sentence translation quality and timing statistics