Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

PARSE: An Open-Domain Reasoning Question Answering Benchmark for Persian

Jamshid Mozafari, Seyed Parsa Mousavinasab, Adam Jatowt

TL;DR

PARSE addresses the lack of open-domain reasoning QA benchmarks for Persian by delivering 10,800 diverse questions across Boolean, MCQ, and factoid formats with varying reasoning demands. It combines a structured taxonomy, an LLM-driven prompting pipeline, and multi-stage quality control to produce high-quality, linguistically robust data, validated by human evaluation. Benchmarks across multilingual and Persian LLMs show Persian prompts and structured prompting (CoT for Boolean/MCQ; few-shot for factoid) enhance performance, and fine-tuning on PARSE yields notable gains, especially for Persian-specialized models like Dorna. The benchmark thus serves both as an evaluation tool and a practical resource for adapting Persian LLMs, with implications for broader multilingual reasoning research and future work in RAG and advanced reasoning models.

Abstract

Reasoning-focused Question Answering (QA) has advanced rapidly with Large Language Models (LLMs), yet high-quality benchmarks for low-resource languages remain scarce. Persian, spoken by roughly 130 million people, lacks a comprehensive open-domain resource for evaluating reasoning-capable QA systems. We introduce PARSE, the first open-domain Persian reasoning QA benchmark, containing 10,800 questions across Boolean, multiple-choice, and factoid formats, with diverse reasoning types, difficulty levels, and answer structures. The benchmark is built via a controlled LLM-based generation pipeline and validated through human evaluation. We also ensure linguistic and factual quality through multi-stage filtering, annotation, and consistency checks. We benchmark multilingual and Persian LLMs under multiple prompting strategies and show that Persian prompts and structured prompting (CoT for Boolean/multiple-choice; few-shot for factoid) improve performance. Fine-tuning further boosts results, especially for Persian-specialized models. These findings highlight how PARSE supports both fair comparison and practical model adaptation. PARSE fills a critical gap in Persian QA research and provides a strong foundation for developing and evaluating reasoning-capable LLMs in low-resource settings.

PARSE: An Open-Domain Reasoning Question Answering Benchmark for Persian

TL;DR

PARSE addresses the lack of open-domain reasoning QA benchmarks for Persian by delivering 10,800 diverse questions across Boolean, MCQ, and factoid formats with varying reasoning demands. It combines a structured taxonomy, an LLM-driven prompting pipeline, and multi-stage quality control to produce high-quality, linguistically robust data, validated by human evaluation. Benchmarks across multilingual and Persian LLMs show Persian prompts and structured prompting (CoT for Boolean/MCQ; few-shot for factoid) enhance performance, and fine-tuning on PARSE yields notable gains, especially for Persian-specialized models like Dorna. The benchmark thus serves both as an evaluation tool and a practical resource for adapting Persian LLMs, with implications for broader multilingual reasoning research and future work in RAG and advanced reasoning models.

Abstract

Reasoning-focused Question Answering (QA) has advanced rapidly with Large Language Models (LLMs), yet high-quality benchmarks for low-resource languages remain scarce. Persian, spoken by roughly 130 million people, lacks a comprehensive open-domain resource for evaluating reasoning-capable QA systems. We introduce PARSE, the first open-domain Persian reasoning QA benchmark, containing 10,800 questions across Boolean, multiple-choice, and factoid formats, with diverse reasoning types, difficulty levels, and answer structures. The benchmark is built via a controlled LLM-based generation pipeline and validated through human evaluation. We also ensure linguistic and factual quality through multi-stage filtering, annotation, and consistency checks. We benchmark multilingual and Persian LLMs under multiple prompting strategies and show that Persian prompts and structured prompting (CoT for Boolean/multiple-choice; few-shot for factoid) improve performance. Fine-tuning further boosts results, especially for Persian-specialized models. These findings highlight how PARSE supports both fair comparison and practical model adaptation. PARSE fills a critical gap in Persian QA research and provides a strong foundation for developing and evaluating reasoning-capable LLMs in low-resource settings.
Paper Structure (15 sections, 3 figures, 6 tables)

This paper contains 15 sections, 3 figures, 6 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Parse Benchmark
  4. Task design and taxonomy.
  5. Prompting pipeline.
  6. Assembly and normalization.
  7. Quality control and balancing.
  8. Human Evaluation
  9. Quality evaluation.
  10. Difficulty evaluation.
  11. Experimental Setup
  12. Experiments
  13. Model performance.
  14. Fine-tuning.
  15. Conclusion

Figures (3)

  • Figure 1: Sample questions from Parse. The English column provides translations of the corresponding Persian questions. Gray-highlighted options indicate the correct answers.
  • Figure 2: Human evaluation accuracy across difficulty levels (easy, medium, hard) for different question types.
  • Figure 3: Performance of LLaMA 3 8B and Dorna before and after fine-tuning on Parse. The figure reports evaluation scores on the 2,160-item test set sampled across all configurations.