Retrieval-Augmented Generation for Service Discovery: Chunking Strategies and Benchmarking

TL;DR

This paper tackles token-length constraints in OpenAPI-driven service discovery by proposing a Retrieval-Augmented Generation (RAG) framework with OpenAPI chunking and an on-demand Discovery Agent. It introduces SOCBench-D, a broad, domain-spanning benchmark, to evaluate endpoint retrieval across multiple domains, and validates the approach with RestBench for real-world applicability. The study shows that endpoint-based, token-efficient chunking generally outperforms whole-document methods and that a Discovery Agent can improve precision at the potential cost of recall, highlighting a trade-off between prompt size and retrieval completeness. Together, the OpenAPI RAG and Discovery Agent offer a viable, scalable path to automatic service composition under context size limits, with practical guidance on chunking strategies and agent-enabled retrieval for real-world deployments.

Abstract

Integrating multiple (sub-)systems is essential to create advanced Information Systems. Difficulties mainly arise when integrating dynamic environments, e.g., the integration at design time of not yet existing services. This has been traditionally addressed using a registry that provides the API documentation of the endpoints. Large Language Models have shown to be capable of automatically creating system integrations (e.g., as service composition) based on this documentation but require concise input due to input oken limitations, especially regarding comprehensive API descriptions. Currently, it is unknown how best to preprocess these API descriptions. In the present work, we (i) analyze the usage of Retrieval Augmented Generation for endpoint discovery and the chunking, i.e., preprocessing, of state-of-practice OpenAPIs to reduce the input oken length while preserving the most relevant information. To further reduce the input token length for the composition prompt and improve endpoint retrieval, we propose (ii) a Discovery Agent that only receives a summary of the most relevant endpoints nd retrieves specification details on demand. We evaluate RAG for endpoint discovery using (iii) a proposed novel service discovery benchmark SOCBench-D representing a general setting across numerous domains and the real-world RestBench enchmark, first, for the different chunking possibilities and parameters measuring the endpoint retrieval accuracy. Then, we assess the Discovery Agent using the same test data set. The prototype shows how to successfully employ RAG for endpoint discovery to reduce the token count. Our experiments show that endpoint-based approaches outperform naive chunking methods for preprocessing. Relying on an agent significantly improves precision while being prone to decrease recall, disclosing the need for further reasoning capabilities.

Figures (10)

• Figure 1: rag for Endpoint Discovery
• Figure 2: Overview of the Discovery Agent Approach for Endpoint Discovery
• Figure 3: Cross-Domain Average Analysis
• Figure 4: Recall by Chunking Strategy as Boxplots Grouped by Model for $k=20$. Model Color-Coded.
• Figure 5: Statistical Stability Analysis of the Candidates.