Retrieval-Augmented Generation for Reliable Interpretation of Radio Regulations

TL;DR

The paper addresses the risk of hallucinations in regulatory QA within telecommunications by introducing a domain-specific Retrieval-Augmented Generation (RAG) pipeline and releasing the first MCQ benchmark derived from ITU Radio Regulations. It combines a FAISS-based retrieval system with dense embeddings and a generation module to ground answers in authoritative text, achieving substantial improvements over naïve prompting (e.g., up to +11.9 percentage points for GPT-4o). A domain-targeted evaluation framework accompanies the dataset, alongside an end-to-end MCQ accuracy measure and a retrieval-focused metric to isolate retrieval quality. The results demonstrate that carefully structured grounding significantly enhances reliability in high-stakes regulatory interpretation, and the deployed Radio Regulations GPT showcases practical applicability and updatability as regulations evolve.

Abstract

We study question answering in the domain of radio regulations, a legally sensitive and high-stakes area. We propose a telecom-specific Retrieval-Augmented Generation (RAG) pipeline and introduce, to our knowledge, the first multiple-choice evaluation set for this domain, constructed from authoritative sources using automated filtering and human validation. To assess retrieval quality, we define a domain-specific retrieval metric, under which our retriever achieves approximately 97% accuracy. Beyond retrieval, our approach consistently improves generation accuracy across all tested models. In particular, while naively inserting documents without structured retrieval yields only marginal gains for GPT-4o (less than 1%), applying our pipeline results in nearly a 12% relative improvement. These findings demonstrate that carefully targeted grounding provides a simple yet strong baseline and an effective domain-specific solution for regulatory question answering. All code and evaluation scripts, along with our derived question-answer dataset, are available at https://github.com/Zakaria010/Radio-RAG.

Figures (7)

• Figure 1: Distribution (word cloud) of key terms in the Radio Regulations corpus, highlighting the domain-specific vocabulary our pipeline must handle
• Figure 2: Overview of our Retrieval-Augmented Generation (RAG) pipeline for radio regulations QA, combining FAISS-based retrieval with LLM-based answer generation
• Figure 3: Automated pipeline for generating and validating multiple-choice questions from radio regulations, integrating LLM generation, automated judging, and human verification
• Figure 4: Accuracy comparison of vanilla LLMs versus our RAG-augmented approach, showing consistent gains across models, with GPT-4o achieving the largest improvement
• Figure 5: Qualitative comparison of vanilla GPT-4o versus our RAG-augmented approach on a regulatory question, where RAG retrieves the rule and yields the correct answer.