Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

ALBA: A European Portuguese Benchmark for Evaluating Language and Linguistic Dimensions in Generative LLMs

Inês Vieira, Inês Calvo, Iago Paulo, James Furtado, Rafael Ferreira, Diogo Tavares, Diogo Glória-Silva, David Semedo, João Magalhães

Abstract

As Large Language Models (LLMs) expand across multilingual domains, evaluating their performance in under-represented languages becomes increasingly important. European Portuguese (pt-PT) is particularly affected, as existing training data and benchmarks are mainly in Brazilian Portuguese (pt-BR). To address this, we introduce ALBA, a linguistically grounded benchmark designed from the ground up to assess LLM proficiency in linguistic-related tasks in pt-PT across eight linguistic dimensions, including Language Variety, Culture-bound Semantics, Discourse Analysis, Word Plays, Syntax, Morphology, Lexicology, and Phonetics and Phonology. ALBA is manually constructed by language experts and paired with an LLM-as-a-judge framework for scalable evaluation of pt-PT generated language. Experiments on a diverse set of models reveal performance variability across linguistic dimensions, highlighting the need for comprehensive, variety-sensitive benchmarks that support further development of tools in pt-PT.

ALBA: A European Portuguese Benchmark for Evaluating Language and Linguistic Dimensions in Generative LLMs

Abstract

As Large Language Models (LLMs) expand across multilingual domains, evaluating their performance in under-represented languages becomes increasingly important. European Portuguese (pt-PT) is particularly affected, as existing training data and benchmarks are mainly in Brazilian Portuguese (pt-BR). To address this, we introduce ALBA, a linguistically grounded benchmark designed from the ground up to assess LLM proficiency in linguistic-related tasks in pt-PT across eight linguistic dimensions, including Language Variety, Culture-bound Semantics, Discourse Analysis, Word Plays, Syntax, Morphology, Lexicology, and Phonetics and Phonology. ALBA is manually constructed by language experts and paired with an LLM-as-a-judge framework for scalable evaluation of pt-PT generated language. Experiments on a diverse set of models reveal performance variability across linguistic dimensions, highlighting the need for comprehensive, variety-sensitive benchmarks that support further development of tools in pt-PT.

Paper Structure

This paper contains 33 sections, 4 figures, 3 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Language Evaluation Benchmarks.
  4. Linguistics Benchmarks.
  5. European Portuguese Benchmarks.
  6. ALBA Benchmark Dataset
  7. Methodology
  8. Expert-based Reference Questions.
  9. Expert-based Reference Answers.
  10. Linguistic Dimensions
  11. Language Variety
  12. Culture-bound Semantics
  13. Discourse Analysis
  14. Word Plays
  15. Syntax
  16. ...and 18 more sections

Figures (4)

  • Figure 1: ALBA allows the assessment of LLM generative capabilities across eight linguistic dimensions.
  • Figure 2: Top five judge configurations ranked by MAE on the validation set.
  • Figure 3: MAE of different LLM judges on the evaluation set using the optimal configuration.
  • Figure 4: Illustrative answers and judge evaluations from Ministral on various ALBA dimensions.