Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

RAGTurk: Best Practices for Retrieval Augmented Generation in Turkish

Süha Kağan Köse, Mehmet Can Baytekin, Burak Aktaş, Bilge Kaan Görür, Evren Ayberk Munis, Deniz Yılmaz, Muhammed Yusuf Kartal, Çağrı Toraman

TL;DR

This work addresses the lack of non-English RAG benchmarks by constructing a two-part Turkish RAG dataset from Turkish Wikipedia and CulturaX and performing an end-to-end evaluation of seven pipeline stages. It introduces a modular RAG design space and a budgeted genetic algorithm to identify high-performing configurations without extensive tuning, finding that HyDE-based pipelines achieve the highest accuracy but at substantial cost, while a Pareto-optimal configuration using cross-encoder reranking and local context augmentation offers strong performance with lower resource use. The study provides actionable best-practice recommendations tailored to Turkish, including production-friendly and high-accuracy recipes, and emphasizes the importance of balancing retrieval quality with generation fidelity in morphologically rich languages. The authors release datasets, prompts, configurations, and evaluation scripts to support reproducible Turkish-RAG research and pave the way for broader non-English RAG benchmarking.

Abstract

Retrieval-Augmented Generation (RAG) enhances LLM factuality, yet design guidance remains English-centric, limiting insights for morphologically rich languages like Turkish. We address this by constructing a comprehensive Turkish RAG dataset derived from Turkish Wikipedia and CulturaX, comprising question-answer pairs and relevant passage chunks. We benchmark seven stages of the RAG pipeline, from query transformation and reranking to answer refinement, without task-specific fine-tuning. Our results show that complex methods like HyDE maximize accuracy (85%) that is considerably higher than the baseline (78.70%). Also a Pareto-optimal configuration using Cross-encoder Reranking and Context Augmentation achieves comparable performance (84.60%) with much lower cost. We further demonstrate that over-stacking generative modules can degrade performance by distorting morphological cues, whereas simple query clarification with robust reranking offers an effective solution.

RAGTurk: Best Practices for Retrieval Augmented Generation in Turkish

TL;DR

This work addresses the lack of non-English RAG benchmarks by constructing a two-part Turkish RAG dataset from Turkish Wikipedia and CulturaX and performing an end-to-end evaluation of seven pipeline stages. It introduces a modular RAG design space and a budgeted genetic algorithm to identify high-performing configurations without extensive tuning, finding that HyDE-based pipelines achieve the highest accuracy but at substantial cost, while a Pareto-optimal configuration using cross-encoder reranking and local context augmentation offers strong performance with lower resource use. The study provides actionable best-practice recommendations tailored to Turkish, including production-friendly and high-accuracy recipes, and emphasizes the importance of balancing retrieval quality with generation fidelity in morphologically rich languages. The authors release datasets, prompts, configurations, and evaluation scripts to support reproducible Turkish-RAG research and pave the way for broader non-English RAG benchmarking.

Abstract

Retrieval-Augmented Generation (RAG) enhances LLM factuality, yet design guidance remains English-centric, limiting insights for morphologically rich languages like Turkish. We address this by constructing a comprehensive Turkish RAG dataset derived from Turkish Wikipedia and CulturaX, comprising question-answer pairs and relevant passage chunks. We benchmark seven stages of the RAG pipeline, from query transformation and reranking to answer refinement, without task-specific fine-tuning. Our results show that complex methods like HyDE maximize accuracy (85%) that is considerably higher than the baseline (78.70%). Also a Pareto-optimal configuration using Cross-encoder Reranking and Context Augmentation achieves comparable performance (84.60%) with much lower cost. We further demonstrate that over-stacking generative modules can degrade performance by distorting morphological cues, whereas simple query clarification with robust reranking offers an effective solution.
Paper Structure (67 sections, 4 equations, 1 figure, 6 tables, 1 algorithm)

This paper contains 67 sections, 4 equations, 1 figure, 6 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Related Work
  3. Retriever Architectures and Query Transformation.
  4. Candidate Re-ranking, Context Selection, and Condensation.
  5. Answer Refinement and Self-Reflective RAG.
  6. System-Level Optimization: AutoRAG, DSPy-RAG, and GraphRAG.
  7. RAG and Retrieval in Turkish.
  8. Positioning our work.
  9. Dataset Construction
  10. Corpus Acquisition and Filtering
  11. Prompt design for filtering.
  12. Final Web corpus.
  13. Website Frequency Analysis.
  14. Wikipedia Articles.
  15. Header-Aware Chunking
  16. ...and 52 more sections

Figures (1)

  • Figure 1: Overview of the RAG design space.