Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

From RAG to Agentic RAG for Faithful Islamic Question Answering

Gagan Bhatia, Hamdy Mubarak, Mustafa Jarrar, George Mikros, Fadi Zaraket, Mahmoud Alhirthani, Mutaz Al-Khatib, Logan Cochrane, Kareem Darwish, Rashid Yahiaoui, Firoj Alam

TL;DR

This work tackles hallucination and abstention in Islamic question answering by introducing IslamicFaithQA, a bilingual benchmark with 3,810 Arabic–English questions and atomic single-gold answers. It pairs this with an end-to-end data suite (25K Arabic text-grounded SFT pairs, 5K bilingual RL samples, and a Qur'an ayat retrieval corpus of 6,236 units) and proposes an agentic RAG framework that uses structured tool calls to iteratively seek evidence and revise answers. Across Arabic-centric and multilingual LLMs, retrieval consistently improves correctness, and agentic RAG yields the largest gains, achieving state-of-the-art performance and reducing Arabic–English gaps even for smaller models like Qwen3-4B-2507 and Fanar variants. The authors release the experimental resources and datasets to the community, highlighting the practical potential of grounding and tool-augmented reasoning for trustworthy Islamic assistants while noting limitations related to madhahib diversity and potential citation manipulation.

Abstract

LLMs are increasingly used for Islamic question answering, where ungrounded responses may carry serious religious consequences. Yet standard MCQ/MRC-style evaluations do not capture key real-world failure modes, notably free-form hallucinations and whether models appropriately abstain when evidence is lacking. To shed a light on this aspect we introduce ISLAMICFAITHQA, a 3,810-item bilingual (Arabic/English) generative benchmark with atomic single-gold answers, which enables direct measurement of hallucination and abstention. We additionally developed an end-to-end grounded Islamic modelling suite consisting of (i) 25K Arabic text-grounded SFT reasoning pairs, (ii) 5K bilingual preference samples for reward-guided alignment, and (iii) a verse-level Qur'an retrieval corpus of $\sim$6k atomic verses (ayat). Building on these resources, we develop an agentic Quran-grounding framework (agentic RAG) that uses structured tool calls for iterative evidence seeking and answer revision. Experiments across Arabic-centric and multilingual LLMs show that retrieval improves correctness and that agentic RAG yields the largest gains beyond standard RAG, achieving state-of-the-art performance and stronger Arabic-English robustness even with a small model (i.e., Qwen3 4B). We will make the experimental resources and datasets publicly available for the community.

From RAG to Agentic RAG for Faithful Islamic Question Answering

TL;DR

This work tackles hallucination and abstention in Islamic question answering by introducing IslamicFaithQA, a bilingual benchmark with 3,810 Arabic–English questions and atomic single-gold answers. It pairs this with an end-to-end data suite (25K Arabic text-grounded SFT pairs, 5K bilingual RL samples, and a Qur'an ayat retrieval corpus of 6,236 units) and proposes an agentic RAG framework that uses structured tool calls to iteratively seek evidence and revise answers. Across Arabic-centric and multilingual LLMs, retrieval consistently improves correctness, and agentic RAG yields the largest gains, achieving state-of-the-art performance and reducing Arabic–English gaps even for smaller models like Qwen3-4B-2507 and Fanar variants. The authors release the experimental resources and datasets to the community, highlighting the practical potential of grounding and tool-augmented reasoning for trustworthy Islamic assistants while noting limitations related to madhahib diversity and potential citation manipulation.

Abstract

LLMs are increasingly used for Islamic question answering, where ungrounded responses may carry serious religious consequences. Yet standard MCQ/MRC-style evaluations do not capture key real-world failure modes, notably free-form hallucinations and whether models appropriately abstain when evidence is lacking. To shed a light on this aspect we introduce ISLAMICFAITHQA, a 3,810-item bilingual (Arabic/English) generative benchmark with atomic single-gold answers, which enables direct measurement of hallucination and abstention. We additionally developed an end-to-end grounded Islamic modelling suite consisting of (i) 25K Arabic text-grounded SFT reasoning pairs, (ii) 5K bilingual preference samples for reward-guided alignment, and (iii) a verse-level Qur'an retrieval corpus of 6k atomic verses (ayat). Building on these resources, we develop an agentic Quran-grounding framework (agentic RAG) that uses structured tool calls for iterative evidence seeking and answer revision. Experiments across Arabic-centric and multilingual LLMs show that retrieval improves correctness and that agentic RAG yields the largest gains beyond standard RAG, achieving state-of-the-art performance and stronger Arabic-English robustness even with a small model (i.e., Qwen3 4B). We will make the experimental resources and datasets publicly available for the community.
Paper Structure (59 sections, 13 figures, 7 tables)

This paper contains 59 sections, 13 figures, 7 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Benchmarking in Islamic Domain
  4. Factuality in Knowledge-Intensive Domains
  5. Datasets
  6. Training and Alignment Resources
  7. The IslamicFaithQA Benchmark
  8. IslamicFaithQA Curation Pipeline
  9. Diversity and Complexity Analysis.
  10. Methodology
  11. Experimental pipeline.
  12. Models:
  13. SFT for text-grounded reasoning.
  14. Group-Optimized RL Alignment (GSPO)
  15. RAG and Agentic RAG
  16. ...and 44 more sections

Figures (13)

  • Figure 1: Current--Proposed--Outcome.(a) Current Islamic QA. (b) We combine IslamicFaithQA, LLM judging, Quran retrieval, and agentic evidence seeking. (c) This yields more faithful, citation-backed responses.
  • Figure 2: The construction pipeline for IslamicFaithQA.
  • Figure 3: Statistical analysis of IslamicFaithQA. (a) Difficulty Distribution: The dataset exhibits a balanced spread across five difficulty levels. (b) Topic Diversity: The benchmark covers a wide range of theological domains.
  • Figure 4: End-to-end development and evaluation workflow.
  • Figure 5: Example item from IslamicFaithQA (Islamic Jurisprudence). We show the query, the atomic gold target, and a model prediction. The LLM-as-a-judge assigns a discrete label (Correct/Incorrect/Not_Attempted) based on semantic equivalence to the single-gold answer.
  • ...and 8 more figures