A RAG-Based Institutional Assistant

TL;DR

The paper addresses the challenge that large language models struggle with knowledge-intensive tasks without access to structured sources. It proposes a modular retrieval-augmented generation (RAG) system tailored to the University of São Paulo, combining a retriever over 866 normative documents with a generator, and evaluating via a generated QA dataset (592 questions) and paraphrased variants. Key findings show that the retriever alone achieves a Top-5 accuracy of 30% and the best generator attains 22.04% accuracy against ground truth, but providing the correct document chunks to the LLM markedly increases accuracy to 54.02%, while no-context prompts drop to 13.68%. The work highlights the critical role of database access for LLM performance, exposes limitations of current semantic search in precise document retrieval, and provides a runnable RAG framework with open data/code to support institutional information tasks.

Abstract

Although large language models (LLMs) demonstrate strong text generation capabilities, they struggle in scenarios requiring access to structured knowledge bases or specific documents, limiting their effectiveness in knowledge-intensive tasks. To address this limitation, retrieval-augmented generation (RAG) models have been developed, enabling generative models to incorporate relevant document fragments into their inputs. In this paper, we design and evaluate a RAG-based virtual assistant specifically tailored for the University of São Paulo. Our system architecture comprises two key modules: a retriever and a generative model. We experiment with different types of models for both components, adjusting hyperparameters such as chunk size and the number of retrieved documents. Our optimal retriever model achieves a Top-5 accuracy of 30%, while our most effective generative model scores 22.04\% against ground truth answers. Notably, when the correct document chunks are supplied to the LLMs, accuracy significantly improves to 54.02%, an increase of over 30 percentage points. Conversely, without contextual input, performance declines to 13.68%. These findings highlight the critical role of database access in enhancing LLM performance. They also reveal the limitations of current semantic search methods in accurately identifying relevant documents and underscore the ongoing challenges LLMs face in generating precise responses.

Figures (5)

• Figure 1: Example from the QA dataset, with question, answer, and context.
• Figure 2: Overview of the system's architecture.
• Figure 3: Top-k accuracy for retriever models using the original questions. Different models are represented by color, and chunk sizes are distinguished by line type.
• Figure 4: Top-k accuracy for retriever models using the paraphrased questions. Different models are represented by color, and chunk sizes are distinguished by line type.
• Figure 5: Correlation matrix showing the relationships between different evaluation metrics (F1, cosine similarity, LLM, and human assessment).