Physical Commonsense Reasoning for Lower-Resourced Languages and Dialects: a Study on Basque

TL;DR

This study examines physical commonsense reasoning in Basque by introducing BasPhyCo, a non-QA dataset derived from the Italian GITA and adapted to Standard Basque and a Western dialect (BasPhyCowest). It evaluates multilingual and language-specific LLMs on three hierarchical tasks—story plausibility (accuracy), conflict detection (consistency), and physical-state verifiability—across Italian, Standard Basque, and Western Basque. The findings show that verifiability is notably weak in Basque, with dialectal variation further reducing performance, though Basque-target pretraining (e.g., Latxa) can improve robustness and state-level reasoning. The work establishes a baseline for Basque physical commonsense evaluation, highlights the need for dialect-aware models and larger resources, and points to future work extending to additional languages and dialects.

Abstract

Physical commonsense reasoning represents a fundamental capability of human intelligence, enabling individuals to understand their environment, predict future events, and navigate physical spaces. Recent years have witnessed growing interest in reasoning tasks within Natural Language Processing (NLP). However, no prior research has examined the performance of Large Language Models (LLMs) on non-question-answering (non-QA) physical commonsense reasoning tasks in low-resource languages such as Basque. Taking the Italian GITA as a starting point, this paper addresses this gap by presenting BasPhyCo, the first non-QA physical commonsense reasoning dataset for Basque, available in both standard and dialectal variants. We evaluate model performance across three hierarchical levels of commonsense understanding: (1) distinguishing between plausible and implausible narratives (accuracy), (2) identifying the conflicting element that renders a narrative implausible (consistency), and (3) determining the specific physical state that creates the implausibility (verifiability). These tasks were assessed using multiple multilingual LLMs as well as models pretrained specifically for Italian and Basque. Results indicate that, in terms of verifiability, LLMs exhibit limited physical commonsense capabilities in low-resource languages such as Basque, especially when processing dialectal variants.

Figures (1)

• Figure 1: The number of unique words in BasPhyCo (left) and BasPhyCowest (right), as well as the overlap of both datasets (middle). Additionally, some examples from each dataset.