MALAMUTE: A Multilingual, Highly-granular, Template-free, Education-based Probing Dataset

TL;DR

MALAMUTE addresses the need for targeted evaluation of language models in education by introducing a multilingual, template-free cloze-style probing dataset derived from OpenStax textbooks across English, Spanish, and Polish. It delivers eight domains with up to 14 subdomains and 33,361 concepts, totaling 116,887 prompts that include both sentence-level and paragraph-level prompts. The authors evaluate eight causal LMs and five MLMs, exploring three prompt designs and demonstrating that granular, context-rich prompts reveal substantial knowledge gaps and language-dependent performance. The dataset supports safer, more effective deployment of AI in classrooms and sets a foundation for future, finer-grained educational benchmarks and model improvements.

Abstract

Language models (LMs) have excelled in various broad domains. However, to ensure their safe and effective integration into real-world educational settings, they must demonstrate proficiency in specific, granular areas of knowledge. Existing cloze-style benchmarks, commonly used to evaluate LMs' knowledge, have three major limitations. They: 1) do not cover the educational domain; 2) typically focus on low-complexity, generic knowledge or broad domains, which do not adequately assess the models' knowledge in specific subjects; and 3) often rely on templates that can bias model predictions. Here, we introduce MALAMUTE, a multilingual, template-free, and highly granular probing dataset comprising expert-written, peer-reviewed probes from 71 university-level textbooks across three languages (English, Spanish, and Polish). MALAMUTE is the first education-based cloze-style dataset. It covers eight domains, each with up to 14 subdomains, further broken down into concepts and concept-based prompts, totaling 33,361 university curriculum concepts and 116,887 prompts. MALAMUTE's fine granularity, educational focus, and inclusion of both sentence-level and paragraph-level prompts make it an ideal tool for evaluating LMs' course-related knowledge. Our evaluation of masked and causal LMs on MALAMUTE shows that despite overall proficiency, they have significant gaps in knowledge when examined closely on specific subjects, hindering their safe use in classrooms and underscoring the need for further development.

Figures (1)

• Figure 1: MALAMUTE: a highly granular cloze-style dataset for educational knowledge probing. This dataset assesses language models' knowledge across three languages with fine-grained detail. Bold strings indicate predictable text. Example prompts shown for 3 out of the 55 subdomains.