PolicyBot - Reliable Question Answering over Policy Documents

TL;DR

PolicyBot tackles the challenge of answering questions over dense policy documents by deploying a domain-specific retrieval-augmented generation pipeline that emphasizes grounded, source-cited responses. It combines semantic chunking, multilingual dense embeddings, HyDE-inspired retrieval, reciprocal rank fusion, and cross-encoder reranking to achieve accurate, transparent answers, all deployable offline on consumer hardware. The system prioritizes factual grounding through direct quotations, explicit source chunks, and a not-enough-context option when context is missing, supported by citation tracing in the UI. The work demonstrates that a carefully engineered, open-source, hardware-efficient RAG stack can improve accessibility to governance information while maintaining privacy, reproducibility, and user trust, and it outlines concrete directions for multilingual expansion, stronger hallucination controls, and user-centered evaluation.

Abstract

All citizens of a country are affected by the laws and policies introduced by their government. These laws and policies serve essential functions for citizens. Such as granting them certain rights or imposing specific obligations. However, these documents are often lengthy, complex, and difficult to navigate, making it challenging for citizens to locate and understand relevant information. This work presents PolicyBot, a retrieval-augmented generation (RAG) system designed to answer user queries over policy documents with a focus on transparency and reproducibility. The system combines domain-specific semantic chunking, multilingual dense embeddings, multi-stage retrieval with reranking, and source-aware generation to provide responses grounded in the original documents. We implemented citation tracing to reduce hallucinations and improve user trust, and evaluated alternative retrieval and generation configurations to identify effective design choices. The end-to-end pipeline is built entirely with open-source tools, enabling easy adaptation to other domains requiring document-grounded question answering. This work highlights design considerations, practical challenges, and lessons learned in deploying trustworthy RAG systems for governance-related contexts.

Figures (3)

• Figure 1: Architecture of the PolicyBot system.
• Figure 2: Comparative performance of embedding models at various retrieval depths
• Figure 3: Comparison of chunking strategies at various retrieval depths