Macaron: Controlled, Human-Written Benchmark for Multilingual and Multicultural Reasoning via Template-Filling

TL;DR

Macaron addresses the gap in multilingual benchmarks by introducing a template-first framework that decouples reasoning type from cultural grounding. It constructs 100 language-agnostic templates spanning 7 reasoning types and 22 cultural aspects to generate scenario-aligned English and local-language MCQs plus derived True/False items, resulting in 11,862 evaluation instances across 20 languages and 20 cultural contexts. The study evaluates 21 multilingual LLMs in zero-shot settings, showing that reasoning-focused models achieve strong, near-language-robust performance, while open-weight models lag notably in local languages and in binary verification tasks, with culture-grounded math posing the greatest challenge. Macaron enables fine-grained diagnostic evaluation of cultural robustness and reasoning, and its template-based design supports scalable extension to new cultures with consistent structure and difficulty. The benchmark is publicly accessible and intended as a diagnostic tool to guide culturally aware model development, while acknowledging coarse cultural coverage and controlled task formats.

Abstract

Multilingual benchmarks rarely test reasoning over culturally grounded premises: translated datasets keep English-centric scenarios, while culture-first datasets often lack control over the reasoning required. We propose Macaron, a template-first benchmark that factorizes reasoning type and cultural aspect across question languages. Using 100 language-agnostic templates that cover 7 reasoning types, 22 cultural aspects, native annotators create scenario-aligned English and local-language multiple-choice questions and systematically derived True/False questions. Macaron contains 11,862 instances spanning 20 countries/cultural contexts, 10 scripts, and 20 languages (including low-resource ones like Amharic, Yoruba, Zulu, Kyrgyz, and some Arabic dialects). In zero-shot evaluation of 21 multilingual LLMs, reasoning-mode models achieve the strongest performance and near-parity between English and local languages, while open-weight models degrade substantially in local languages and often approach chance on T/F tasks. Culture-grounded mathematical and counting templates are consistently the hardest. The data can be accessed here https://huggingface.co/datasets/AlaaAhmed2444/Macaron.

Figures (12)

• Figure 1: Macaron Coverage. Our benchmark spans 20 countries, 20 languages, 10 scripts; each mapped country (colored in blue) contributes scenario-aligned English and local-language items based on the predesigned set of templates.
• Figure 2: Macaron Curation Pipeline. We first design language-agnostic templates tagged with reasoning categories and cultural aspects. Native annotators instantiate each template with culturally grounded content to produce scenario-aligned English MCQs and translate them into local languages. From each MCQ, we derive aligned True/False statements by instantiating the template’s binary child forms with the correct option (True) and a distractor (False), and translate these statements into the local language. Finally, we run an LLM proofreading pass to correct minor spelling/grammar issues while preserving the original meaning and leaving all local-language text unchanged.
• Figure 3: Average performance in English vs. local language across cultural contexts.
• Figure 4: Accuracy by reasoning type across models.
• Figure 5: Accuracy (%) by cultural aspect across models. Each cell aggregates over evaluation instances whose templates are tagged with the corresponding aspect (multi-label; a single instance may contribute to multiple aspects). Aspects are ordered by mean accuracy (hard $\rightarrow$ easy).