BenchMAX: A Comprehensive Multilingual Evaluation Suite for Large Language Models

TL;DR

BenchMAX addresses the gap in evaluating advanced, language-agnostic capabilities of large language models across 17 languages. It introduces a rigorous three-step translation and annotation pipeline to create high-quality multilingual data and evaluates a broad set of models on 6 capabilities across 10 tasks. The results show that increasing model size improves overall multilingual performance but fails to eradicate the English–non-English gaps, and domain-specific translation presents new evaluation challenges. The work demonstrates BenchMAX as a valuable, publicly available test bed for advancing multilingual instruction following, reasoning, and code generation, with insights into open-source versus closed-source model behavior.

Abstract

Previous multilingual benchmarks focus primarily on simple understanding tasks, but for large language models(LLMs), we emphasize proficiency in instruction following, reasoning, long context understanding, code generation, and so on. However, measuring these advanced capabilities across languages is underexplored. To address the disparity, we introduce BenchMAX, a multi-way multilingual evaluation benchmark that allows for fair comparisons of these important abilities across languages. To maintain high quality, three distinct native-speaking annotators independently annotate each sample within all tasks after the data was machine-translated from English into 16 other languages. Additionally, we present a novel translation challenge stemming from dataset construction. Extensive experiments on BenchMAX reveal varying effectiveness of core capabilities across languages, highlighting performance gaps that cannot be bridged by simply scaling up model size. BenchMAX serves as a comprehensive multilingual evaluation platform, providing a promising test bed to promote the development of multilingual language models. The dataset and code are publicly accessible.

Figures (9)

• Figure 1: BenchMAX evaluates diverse advanced capabilities of LLMs in multilingual context.
• Figure 2: The construction process involves three steps: Step 1) translating data from English to non-English; Step 2) post-editing each sample by three human annotators; Step 3) selecting the final translation version.
• Figure 3: Flow chart illustrating the constraint extraction and machine translation pipeline in the first step of our benchmark construction.
• Figure 4: Taking two tasks as examples, models exhibit unbalanced multilingual capabilities. We show performance of several models on the science reasoning task and the domain translation task across different languages.
• Figure 5: Larger models do not consistently have a smaller GAP. Each row shows proportions of tasks where the larger model achieves a smaller GAP versus where the smaller model performs better.