PISA-Bench: The PISA Index as a Multilingual and Multimodal Metric for the Evaluation of Vision-Language Models

TL;DR

PISA-Bench introduces a multilingual, multimodal benchmark for evaluating vision-language models by transforming expert-created PISA test items into a high-quality dataset with parallel translations across six languages. The authors implement a four-stage dataset construction pipeline (collection, modular extraction, quality control, translation) and validate translations with automatic and human assessments, demonstrating overall translation reliability. Experimental results reveal that state-of-the-art models struggle across languages, with pronounced difficulty in spatial and geometric reasoning and noticeable degradation on non-English splits, while larger models and proprietary systems perform comparatively better. The work also includes contamination analysis to confirm the benchmark’s reliability and provides a Rasch-based mapping to the PISA scale for interpretability. By releasing the dataset and evaluation framework, PISA-Bench aims to drive progress in multilingual multimodal reasoning and fair cross-language evaluation.

Abstract

Vision-language models (VLMs) have demonstrated remarkable progress in multimodal reasoning. However, existing benchmarks remain limited in terms of high-quality, human-verified examples. Many current datasets rely on synthetically generated content by large language models (LLMs). Furthermore, most datasets are limited to English, as manual quality assurance of translated samples is time-consuming and costly. To fill this gap, we introduce PISA-Bench, a multilingual benchmark derived from English examples of the expert-created PISA tests, a unified framework for the assessment of student competencies in over eighty countries. Each example consists of human-extracted instructions, questions, answer options, and images, enriched with question type categories, and has been translated from English into five additional languages (Spanish, German, Chinese, French, and Italian), resulting in a fully parallel corpus covering six languages. We evaluate state-of-the-art vision-language models on PISA-Bench and find that especially small models (<20B parameters) fail to achieve high test scores. We further find substantial performance degradation on non-English splits as well as high error-rates when models are tasked with spatial and geometric reasoning. By releasing the dataset and evaluation framework, we provide a resource for advancing research on multilingual multimodal reasoning.

Figures (5)

• Figure 1: Overview of the dataset construction pipeline. We (1) collect tasks from the original OECD PISA tests, (2) decompose them into modular components (instruction, image, question, and answer options), (3) verify, augment, and, if necessary, correct the extracted content, and (4) translate them into five target languages (ES, DE, CH, FR, IT) and verify translations through native speakers.
• Figure 2: Distribution of WMT23 COMET-KIWI scores for each target language.
• Figure 3: Comparison of error rates across languages for GPT-4o and Qwen-3-VL-8B-Instruct.
• Figure 4: Prompt used for augmenting the original English test questions, e.g., required when multiple-choice answer options are not available or the question is not self-contained.
• Figure 5: Prompt template used for generating translations using GPT-4.