Towards Requirements Engineering for RAG Systems

TL;DR

The paper addresses the challenge of deploying Retrieval Augmented Generation (RAG) systems in expert-domain settings by examining how developers elicit retrieval requirements in a maritime context. Through an in-depth case study at Marcomp, it proposes an empirical, five-stage process—knowledge modeling and experimentation, retrieval strategy, retrievable data management, monitoring and operation, plus ongoing expectation management—to guide the development of RAG systems. Key findings show that data-driven filtering, context sensitivity, and substantial human-in-the-loop verification are essential to maintain correctness amid evolving rules and diverse contexts. The work contributes a practical RE workflow for RAG and offers actionable guidance for data governance and user collaboration in complex domain deployments.

Abstract

This short paper explores how a maritime company develops and integrates large-language models (LLM). Specifically by looking at the requirements engineering for Retrieval Augmented Generation (RAG) systems in expert settings. Through a case study at a maritime service provider, we demonstrate how data scientists face a fundamental tension between user expectations of AI perfection and the correctness of the generated outputs. Our findings reveal that data scientists must identify context-specific "retrieval requirements" through iterative experimentation together with users because they are the ones who can determine correctness. We present an empirical process model describing how data scientists practically elicited these "retrieval requirements" and managed system limitations. This work advances software engineering knowledge by providing insights into the specialized requirements engineering processes for implementing RAG systems in complex domain-specific applications.

Figures (2)

• Figure 1: The RAG system as developed by Marcomp. First, a question—e.g., from a ship's captain—is sent to a case handler. The case handler sets filters to help the RAG system retrieve relevant past answers. These, along with the question, form the input for the LLM. The generated answer is then reviewed and often revised by the case handler before sending a final answer back to the captain.
• Figure 2: An iterative five-stage process model for eliciting "retrieval requirements" (RR). First, data scientists explore the available data in the knowledge base. Second, they define which parts should be retrievable as input to the LLM. Third, they work with users to assess the RAG system’s output and design filtering functions for tailored retrieval. Fourth, they monitor the live system to identify new "retrieval requirements". Fifth, they continuously manage user expectations as the system evolves.