PerCoR: Evaluating Commonsense Reasoning in Persian via Multiple-Choice Sentence Completion

TL;DR

PerCoR addresses the lack of Persian commonsense benchmarks by introducing a 106K Persian multiple-choice sentence completion dataset built from diverse sources. A conjunction-based segmentation scheme enables coherent pair creation, while DRESS-AF provides adversarially filtered, embedding-based distractors without relying on generative models. Evaluations across 32 models show a substantial gap between top closed-source systems and high-quality open models, with human accuracy around 89%, and the dataset challenges even strong models. The work also demonstrates cross-lingual transfer by transferring DRESS-AF to HellaSwag, and outlines a path for multilingual extension and lightweight fine-tuning to close the gap.

Abstract

We introduced PerCoR (Persian Commonsense Reasoning), the first large-scale Persian benchmark for commonsense reasoning. PerCoR contains 106K multiple-choice sentence-completion problems drawn from more than forty news, cultural, and other web sources. We introduce a novel conjunction-based segmentation strategy to generate coherent sentence-completion pairs, enabling broad topical and structural diversity. To create challenging distractors, we propose DRESS-AF (Distractor Ranking via Embedding Similarity Scoring and Adversarial Filtering), a generation-free adversarial filtering method that selects distractors from the pool of gold continuations while maximising model confusion. Human annotators score 89% on PerCoR, while OpenAI-o3 achieves the highest performance at 92.18%, followed closely by Claude-Sonnet-3.7 (91.17%). The strongest open-source model, DeepSeek-R1, reaches 82.51%, underscoring both the dataset's difficulty and the remaining performance gap in Persian commonsense reasoning. We further show that DRESS-AF transfers to the English HellaSwag benchmark, increasing its difficulty without hurting human solvability. The dataset is available at https://huggingface.co/datasets/MCINext/PerCoR.

Figures (17)

• Figure 1: An example from the PerCoR dataset. The passage discusses the pleasant spring weather in Qeshm and recommends nighttime exploration. The correct answer (written in Green) refers to night camps and music breaking the beach's silence, while other options, though plausible in isolation, lack relevance to the immediate context.
• Figure 2: Overview of our dataset construction and distractor generation pipeline. The process consists of: (a) collecting diverse Persian text data, (b) creating and filtering sentence-completion pairs, and (c) generating challenging multiple-choice distractors using DRESS-AF.
• Figure 3: Accuracy of GPT-4o-mini on the provisional dataset, during the construction of the PerCoR dataset. DRESS-AF tries to find the best coefficients within 30 trials. The first 10 trials use random sampling, followed by TPE-based search. The lowest accuracy (trial 20) corresponds to the selected distractor configuration.
• Figure 4: Accuracy of GPT-4o-mini on the provisional dataset across 30 trials during DRESS-AF optimisation on sentence--completion pairs from HellaSwag. The left plot corresponds to the harder version, with distractors sampled from the top 10 candidates. The right plot corresponds to the easier version, where the top 3 candidates are excluded and distractors are sampled from the next top 20.
• Figure 5: Accuracy of GPT-4o-mini and Gemma3-27B-it, representing closed- and open-source models respectively, across three HellaSwag variants. DRESS-AF was used to generate distractors for the easier and harder variants.