Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Chunking, Retrieval, and Re-ranking: An Empirical Evaluation of RAG Architectures for Policy Document Question Answering

Anuj Maharjan, Umesh Yadav

TL;DR

Problem: LLMs can hallucinate when answering policy questions; need grounding in authoritative CDC policy documents. Approach: empirically compare Vanilla LLM, Basic RAG, and Advanced RAG pipelines using dual-stage retrieval on a CDC policy corpus with two chunking strategies. Findings: Basic RAG improves faithfulness over Vanilla (0.621 vs 0.347); Advanced RAG achieves the highest faithfulness (0.797); two-stage retrieval is essential, though fragmentation of policy workflows by chunking remains a bottleneck. Significance: establishes a reproducible baseline for policy-aware intelligent navigators in regulated settings and informs design choices to maximize factual fidelity with current retrieval technology.

Abstract

The integration of Large Language Models (LLMs) into the public health policy sector offers a transformative approach to navigating the vast repositories of regulatory guidance maintained by agencies such as the Centers for Disease Control and Prevention (CDC). However, the propensity for LLMs to generate hallucinations, defined as plausible but factually incorrect assertions, presents a critical barrier to the adoption of these technologies in high-stakes environments where information integrity is non-negotiable. This empirical evaluation explores the effectiveness of Retrieval-Augmented Generation (RAG) architectures in mitigating these risks by grounding generative outputs in authoritative document context. Specifically, this study compares a baseline Vanilla LLM against Basic RAG and Advanced RAG pipelines utilizing cross-encoder re-ranking. The experimental framework employs a Mistral-7B-Instruct-v0.2 model and an all-MiniLM-L6-v2 embedding model to process a corpus of official CDC policy analytical frameworks and guidance documents. The analysis measures the impact of two distinct chunking strategies, recursive character-based and token-based semantic splitting, on system accuracy, measured through faithfulness and relevance scores across a curated set of complex policy scenarios. Quantitative findings indicate that while Basic RAG architectures provide a substantial improvement in faithfulness (0.621) over Vanilla baselines (0.347), the Advanced RAG configuration achieves a superior faithfulness average of 0.797. These results demonstrate that two-stage retrieval mechanisms are essential for achieving the precision required for domain-specific policy question answering, though structural constraints in document segmentation remain a significant bottleneck for multi-step reasoning tasks.

Chunking, Retrieval, and Re-ranking: An Empirical Evaluation of RAG Architectures for Policy Document Question Answering

TL;DR

Problem: LLMs can hallucinate when answering policy questions; need grounding in authoritative CDC policy documents. Approach: empirically compare Vanilla LLM, Basic RAG, and Advanced RAG pipelines using dual-stage retrieval on a CDC policy corpus with two chunking strategies. Findings: Basic RAG improves faithfulness over Vanilla (0.621 vs 0.347); Advanced RAG achieves the highest faithfulness (0.797); two-stage retrieval is essential, though fragmentation of policy workflows by chunking remains a bottleneck. Significance: establishes a reproducible baseline for policy-aware intelligent navigators in regulated settings and informs design choices to maximize factual fidelity with current retrieval technology.

Abstract

The integration of Large Language Models (LLMs) into the public health policy sector offers a transformative approach to navigating the vast repositories of regulatory guidance maintained by agencies such as the Centers for Disease Control and Prevention (CDC). However, the propensity for LLMs to generate hallucinations, defined as plausible but factually incorrect assertions, presents a critical barrier to the adoption of these technologies in high-stakes environments where information integrity is non-negotiable. This empirical evaluation explores the effectiveness of Retrieval-Augmented Generation (RAG) architectures in mitigating these risks by grounding generative outputs in authoritative document context. Specifically, this study compares a baseline Vanilla LLM against Basic RAG and Advanced RAG pipelines utilizing cross-encoder re-ranking. The experimental framework employs a Mistral-7B-Instruct-v0.2 model and an all-MiniLM-L6-v2 embedding model to process a corpus of official CDC policy analytical frameworks and guidance documents. The analysis measures the impact of two distinct chunking strategies, recursive character-based and token-based semantic splitting, on system accuracy, measured through faithfulness and relevance scores across a curated set of complex policy scenarios. Quantitative findings indicate that while Basic RAG architectures provide a substantial improvement in faithfulness (0.621) over Vanilla baselines (0.347), the Advanced RAG configuration achieves a superior faithfulness average of 0.797. These results demonstrate that two-stage retrieval mechanisms are essential for achieving the precision required for domain-specific policy question answering, though structural constraints in document segmentation remain a significant bottleneck for multi-step reasoning tasks.
Paper Structure (15 sections, 2 equations, 2 figures, 3 tables, 1 algorithm)

This paper contains 15 sections, 2 equations, 2 figures, 3 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. State of the Field and Comparative Literature
  3. The Evolution of Retrieval Mechanisms
  4. Legal and Regulatory NLP
  5. Methodology and Architecture
  6. System Architecture
  7. Algorithmic Formulation
  8. Mathematical Formulation
  9. System Configurations
  10. Experimental Results
  11. Quantitative Performance Comparison
  12. Qualitative Case Study
  13. Analysis of System Evolution
  14. Quantitative Impact of Re-ranking Accuracy
  15. Conclusion and Future Work

Figures (2)

  • Figure 1: The Advanced RAG Architecture. The pipeline uses a Bi-Encoder for initial efficient retrieval, followed by a computationally intensive Cross-Encoder to filter false positives before generation.
  • Figure 2: Comparison of Average Faithfulness and Relevance Scores. The Advanced RAG architecture demonstrates superior performance in grounding answers to the source text.