Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Recovered in Translation: Efficient Pipeline for Automated Translation of Benchmarks and Datasets

Hanna Yukhymenko, Anton Alexandrov, Martin Vechev

TL;DR

This work presents a fully automated framework designed to address the reliability of multilingual Large Language Model evaluation by enabling scalable, high-quality translation of datasets and benchmarks, and demonstrates that adapting test-time compute scaling strategies, specifically Universal Self-Improvement (USI) and the proposed multi-round ranking method, T-RANK, allows for significantly higher quality outputs compared to traditional pipelines.

Abstract

The reliability of multilingual Large Language Model (LLM) evaluation is currently compromised by the inconsistent quality of translated benchmarks. Existing resources often suffer from semantic drift and context loss, which can lead to misleading performance metrics. In this work, we present a fully automated framework designed to address these challenges by enabling scalable, high-quality translation of datasets and benchmarks. We demonstrate that adapting test-time compute scaling strategies, specifically Universal Self-Improvement (USI) and our proposed multi-round ranking method, T-RANK, allows for significantly higher quality outputs compared to traditional pipelines. Our framework ensures that benchmarks preserve their original task structure and linguistic nuances during localization. We apply this approach to translate popular benchmarks and datasets into eight Eastern and Southern European languages (Ukrainian, Bulgarian, Slovak, Romanian, Lithuanian, Estonian, Turkish, Greek). Evaluations using both reference-based metrics and LLM-as-a-judge show that our translations surpass existing resources, resulting in more accurate downstream model assessment. We release both the framework and the improved benchmarks to facilitate robust and reproducible multilingual AI development.

Recovered in Translation: Efficient Pipeline for Automated Translation of Benchmarks and Datasets

TL;DR

This work presents a fully automated framework designed to address the reliability of multilingual Large Language Model evaluation by enabling scalable, high-quality translation of datasets and benchmarks, and demonstrates that adapting test-time compute scaling strategies, specifically Universal Self-Improvement (USI) and the proposed multi-round ranking method, T-RANK, allows for significantly higher quality outputs compared to traditional pipelines.

Abstract

The reliability of multilingual Large Language Model (LLM) evaluation is currently compromised by the inconsistent quality of translated benchmarks. Existing resources often suffer from semantic drift and context loss, which can lead to misleading performance metrics. In this work, we present a fully automated framework designed to address these challenges by enabling scalable, high-quality translation of datasets and benchmarks. We demonstrate that adapting test-time compute scaling strategies, specifically Universal Self-Improvement (USI) and our proposed multi-round ranking method, T-RANK, allows for significantly higher quality outputs compared to traditional pipelines. Our framework ensures that benchmarks preserve their original task structure and linguistic nuances during localization. We apply this approach to translate popular benchmarks and datasets into eight Eastern and Southern European languages (Ukrainian, Bulgarian, Slovak, Romanian, Lithuanian, Estonian, Turkish, Greek). Evaluations using both reference-based metrics and LLM-as-a-judge show that our translations surpass existing resources, resulting in more accurate downstream model assessment. We release both the framework and the improved benchmarks to facilitate robust and reproducible multilingual AI development.
Paper Structure (21 sections, 5 figures, 26 tables)

This paper contains 21 sections, 5 figures, 26 tables.

Table of Contents

  1. Introduction
  2. Key challenge: multilingual benchmark quality.
  3. Our work:
  4. Main contributions:
  5. Background and Related Work
  6. Foundation for LLM-Based Efficient Translation
  7. Shortcomings of Existing Multilingual Resources
  8. An Efficient Automated Translation Framework
  9. Motivation
  10. Methodology
  11. Benchmark Translation for Eastern European Languages
  12. Evaluation Results
  13. Machine Translation Benchmarks
  14. Multilingual Benchmark Translation Quality
  15. Conclusion
  16. ...and 6 more sections

Figures (5)

  • Figure 1: Universal Self-Improvement (USI) method workflow outlook
  • Figure 2: Translation Ranking (T-RANK) method workflow outlook
  • Figure 3: Translation Ranking (T-RANK) Correction Example
  • Figure 4: Universal Self-Improvement (USI) Correction Example
  • Figure 5: LLM-as-a-judge Quality Evaluation Prompt