Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Bilingual Evaluation of Language Models on General Knowledge in University Entrance Exams with Minimal Contamination

Eva Sánchez Salido, Roser Morante, Julio Gonzalo, Guillermo Marco, Jorge Carrillo-de-Albornoz, Laura Plaza, Enrique Amigó, Andrés Fernández, Alejandro Benito-Santos, Adrián Ghajari Espinosa, Victor Fresno

TL;DR

This work introduces UNED-ACCESS 2024, a private bilingual dataset of 1003 Spanish university-entrance MCQs (professionally translated to English) designed to minimize contamination. Using a uniform zero-shot protocol across 12 models, the authors show that model performance correlates with size, that language gaps between English and Spanish shrink for stronger models, and that results align closely with a corresponding MMLU subset, suggesting robustness of the evaluation. The dataset enables controlled cross-language benchmarking while addressing data leakage concerns, and the findings highlight the practical impact of model scale on cross-language knowledge and reasoning tasks. Overall, UNED-ACCESS 2024 provides a compact, diverse, and privacy-conscious benchmark that supports reliable comparison of general-knowledge capabilities across languages and disciplines, with implications for future extension and standardization in LLM evaluation.

Abstract

In this article we present UNED-ACCESS 2024, a bilingual dataset that consists of 1003 multiple-choice questions of university entrance level exams in Spanish and English. Questions are originally formulated in Spanish and translated manually into English, and have not ever been publicly released. A selection of current open-source and proprietary models are evaluated in a uniform zero-shot experimental setting both on the UNED-ACCESS 2024 dataset and on an equivalent subset of MMLU questions. Results show that (i) reasoning questions are challenging for models, (ii) smaller models perform worse than larger models and degrade faster in Spanish than in English and (iii) the performance gap between languages is negligible for the best models and grows up to 37% for smaller models. Model ranking on UNED-ACCESS 2024 is almost identical in English and Spanish, and has also a high correlation (0.98 Pearson) with ranking on MMLU, suggesting that a small dataset is sufficiently diverse and representative to measure performance by discipline.

Bilingual Evaluation of Language Models on General Knowledge in University Entrance Exams with Minimal Contamination

TL;DR

This work introduces UNED-ACCESS 2024, a private bilingual dataset of 1003 Spanish university-entrance MCQs (professionally translated to English) designed to minimize contamination. Using a uniform zero-shot protocol across 12 models, the authors show that model performance correlates with size, that language gaps between English and Spanish shrink for stronger models, and that results align closely with a corresponding MMLU subset, suggesting robustness of the evaluation. The dataset enables controlled cross-language benchmarking while addressing data leakage concerns, and the findings highlight the practical impact of model scale on cross-language knowledge and reasoning tasks. Overall, UNED-ACCESS 2024 provides a compact, diverse, and privacy-conscious benchmark that supports reliable comparison of general-knowledge capabilities across languages and disciplines, with implications for future extension and standardization in LLM evaluation.

Abstract

In this article we present UNED-ACCESS 2024, a bilingual dataset that consists of 1003 multiple-choice questions of university entrance level exams in Spanish and English. Questions are originally formulated in Spanish and translated manually into English, and have not ever been publicly released. A selection of current open-source and proprietary models are evaluated in a uniform zero-shot experimental setting both on the UNED-ACCESS 2024 dataset and on an equivalent subset of MMLU questions. Results show that (i) reasoning questions are challenging for models, (ii) smaller models perform worse than larger models and degrade faster in Spanish than in English and (iii) the performance gap between languages is negligible for the best models and grows up to 37% for smaller models. Model ranking on UNED-ACCESS 2024 is almost identical in English and Spanish, and has also a high correlation (0.98 Pearson) with ranking on MMLU, suggesting that a small dataset is sufficiently diverse and representative to measure performance by discipline.
Paper Structure (27 sections, 2 equations, 1 figure, 7 tables)

This paper contains 27 sections, 2 equations, 1 figure, 7 tables.

Table of Contents

  1. Introduction
  2. Related work
  3. LLM benchmarking.
  4. Exams benchmarks.
  5. Multilingual evaluations.
  6. Data contamination.
  7. The UNED-ACCESS 2024 Dataset
  8. Experimental setup
  9. Evaluation metrics.
  10. Results
  11. EN-ES Language gap.
  12. Results by subject.
  13. UNED-ACCESS vs MMLU: public vs private test sets.
  14. Conclusions
  15. UNED-ACCESS 2024: Additional Information
  16. ...and 12 more sections

Figures (1)

  • Figure 1: Distribution of results on UNED-ACCESS 2024 per subject (Cohen's Kappa).