Culturally Grounded Physical Commonsense Reasoning in Italian and English: A Submission to the MRL 2025 Shared Task

TL;DR

This paper tackles the scarcity of multilingual physical commonsense datasets by introducing FormaMentis, an expert-crafted Italian benchmark for physical reasoning that is culturally grounded. The dataset follows a PIQA-like format (prompt with two completions, one correct) and covers three everyday domains—household, cuisine, and entertainment—while providing carefully translated English versions that retain Italian cultural references. It emphasizes stringent data-quality controls through a native-speaker validation protocol and guarantees novelty by requiring manual creation rather than translation from other sources. With 120 samples (40 per domain) and a diverse range of prompt lengths (26–452 words), FormaMentis enables robust cross-lingual evaluation of models on culturally nuanced physical reasoning beyond English, potentially guiding future multilingual and multimodal extensions.

Abstract

This paper presents our submission to the MRL 2025 Shared Task on Multilingual Physical Reasoning Datasets. The objective of the shared task is to create manually-annotated evaluation data in the physical commonsense reasoning domain, for languages other than English, following a format similar to PIQA. Our contribution, FormaMentis, is a novel benchmark for physical commonsense reasoning that is grounded in Italian language and culture. The data samples in FormaMentis are created by expert annotators who are native Italian speakers and are familiar with local customs and norms. The samples are additionally translated into English, while preserving the cultural elements unique to the Italian context.

Figures (4)

• Figure 1: FormaMentis domains. For each, we show an example prompt requiring cultural knowledge. E.g., a graduation wreath is a specific type of wreath worn by Italian university students, frittelle di Carnevale is a local holiday recipe, and Ruzzola is a traditional Italian country sport. Completing these prompts correctly thus requires an understanding of the specific characteristics and rules of each item or practice. It is also worth noting that the English translations aim to preserve these culture-specific characteristics, which involves leaving words in their original Italian form where necessary.
• Figure 2: Sample distribution by total number of words (left) and number of sentences in the prompt (right).
• Figure 3: Sample distribution in FormaMentis by number of words in the prompt.
• Figure 4: Sample distribution in FormaMentis by number of words in a completion. We measure the completion length as the average number of words between both completions in a sample.