Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

SeaEval for Multilingual Foundation Models: From Cross-Lingual Alignment to Cultural Reasoning

Bin Wang, Zhengyuan Liu, Xin Huang, Fangkai Jiao, Yang Ding, AiTi Aw, Nancy F. Chen

TL;DR

SeaEval introduces a comprehensive, multilingual benchmark to evaluate foundation models across language understanding, reasoning, culture, and cross-lingual alignment. It expands beyond accuracy by incorporating instruction sensitivity and cross-lingual consistency, using AC3 as a holistic metric. The study reveals persistent issues like instruction paraphrase sensitivity, exposure biases, and multilingual inconsistencies, and shows uneven multilingual proficiency across models. The benchmark, with 29 datasets including 7 new cultural/cross-lingual datasets, aims to drive improvements in semantic representation and cross-lingual contextualization for future multilingual AI systems.

Abstract

We present SeaEval, a benchmark for multilingual foundation models. In addition to characterizing how these models understand and reason with natural language, we also investigate how well they comprehend cultural practices, nuances, and values. Alongside standard accuracy metrics, we investigate the brittleness of foundation models in the dimensions of semantics and multilinguality. Our analyses span both open-sourced and closed models, leading to empirical results across classic NLP tasks, reasoning, and cultural comprehension. Key findings indicate (1) Most models exhibit varied behavior when given paraphrased instructions. (2) Many models still suffer from exposure bias (e.g., positional bias, majority label bias). (3) For questions rooted in factual, scientific, and commonsense knowledge, consistent responses are expected across multilingual queries that are semantically equivalent. Yet, most models surprisingly demonstrate inconsistent performance on these queries. (4) Multilingually-trained models have not attained "balanced multilingual" capabilities. Our endeavors underscore the need for more generalizable semantic representations and enhanced multilingual contextualization. SeaEval can serve as a launchpad for more thorough investigations and evaluations for multilingual and multicultural scenarios.

SeaEval for Multilingual Foundation Models: From Cross-Lingual Alignment to Cultural Reasoning

TL;DR

SeaEval introduces a comprehensive, multilingual benchmark to evaluate foundation models across language understanding, reasoning, culture, and cross-lingual alignment. It expands beyond accuracy by incorporating instruction sensitivity and cross-lingual consistency, using AC3 as a holistic metric. The study reveals persistent issues like instruction paraphrase sensitivity, exposure biases, and multilingual inconsistencies, and shows uneven multilingual proficiency across models. The benchmark, with 29 datasets including 7 new cultural/cross-lingual datasets, aims to drive improvements in semantic representation and cross-lingual contextualization for future multilingual AI systems.

Abstract

We present SeaEval, a benchmark for multilingual foundation models. In addition to characterizing how these models understand and reason with natural language, we also investigate how well they comprehend cultural practices, nuances, and values. Alongside standard accuracy metrics, we investigate the brittleness of foundation models in the dimensions of semantics and multilinguality. Our analyses span both open-sourced and closed models, leading to empirical results across classic NLP tasks, reasoning, and cultural comprehension. Key findings indicate (1) Most models exhibit varied behavior when given paraphrased instructions. (2) Many models still suffer from exposure bias (e.g., positional bias, majority label bias). (3) For questions rooted in factual, scientific, and commonsense knowledge, consistent responses are expected across multilingual queries that are semantically equivalent. Yet, most models surprisingly demonstrate inconsistent performance on these queries. (4) Multilingually-trained models have not attained "balanced multilingual" capabilities. Our endeavors underscore the need for more generalizable semantic representations and enhanced multilingual contextualization. SeaEval can serve as a launchpad for more thorough investigations and evaluations for multilingual and multicultural scenarios.
Paper Structure (19 sections, 3 equations, 15 figures, 6 tables)

This paper contains 19 sections, 3 equations, 15 figures, 6 tables.

Table of Contents

  1. Introduction
  2. Essential Properties of Multilingual Foundation Models and Benchmarks
  3. Multilingual Foundation Model
  4. Multilingual Benchmarks
  5. SeaEval
  6. Task Selection
  7. Data Curation
  8. Evaluation Protocols
  9. Evaluation Results and Discussion
  10. Conclusions
  11. Related Work
  12. Existing Benchmarks
  13. Foundation Language Models
  14. Selected Models
  15. Cross-Lingual Consistency
  16. ...and 4 more sections

Figures (15)

  • Figure 1: SeaEval for multilingual foundation models. English is represented by the color blue, Chinese by red, and a mix of multiple languages by yellow. SeaEval includes the datasets within the dotted-line circle.
  • Figure 2: Two new evaluation protocols for multilingual foundation models in SeaEval. The performance result is taken from ChatGPT on Cross-LogiQA dataset.
  • Figure 3: Performance on MMLU dataset with paraphrased instruction. Some models show large performance variances with paraphrased instruction templates.
  • Figure 4: Effect on label order. Performance varies when labels are shuffled, revealing inherent label biases.
  • Figure 5: Evaluation results of six representative LLMs on a subset of SeaEval. AC3 and Accuracy scores are reported. The error bar covers performances from five different instruction templates.
  • ...and 10 more figures