Optimizing Retrieval Augmented Generation for Object Constraint Language

TL;DR

This work tackles automating Object Constraint Language (OCL) rule generation for Model-Based Systems Engineering (MBSE) by applying Retrieval-Augmented Generation (RAG). It systematically compares three retrieval strategies—BM25, BERT, and SPLADE—across multiple top-$k$ settings to provide context to a Large Language Model, benchmarking against PathOCL. The results show that BM25 underperforms the no-retrieval baseline, while semantic approaches (BERT and SPLADE) improve generation quality, with SPLADE delivering the strongest gains at low $k$ but diminishing performance as $k$ grows due to noise. The findings emphasize that retrieval configuration, particularly the choice of retriever and the number of retrieved chunks, critically affects generation accuracy and consistency, and that semantic retrieval scales better than graph-based path methods on larger meta-models.

Abstract

The Object Constraint Language (OCL) is essential for defining precise constraints within Model-Based Systems Engineering (MBSE). However, manually writing OCL rules is complex and time-consuming. This study explores the optimization of Retrieval-Augmented Generation (RAG) for automating OCL rule generation, focusing on the impact of different retrieval strategies. We evaluate three retrieval approaches $\unicode{x2013}$ BM25 (lexical-based), BERT-based (semantic retrieval), and SPLADE (sparse-vector retrieval) $\unicode{x2013}$ analyzing their effectiveness in providing relevant context for a large language model. To further assess our approach, we compare and benchmark our retrieval-optimized generation results against PathOCL, a state-of-the-art graph-based method. We directly compare BM25, BERT, and SPLADE retrieval methods with PathOCL to understand how different retrieval methods perform for a unified evaluation framework. Our experimental results, focusing on retrieval-augmented generation, indicate that while retrieval can enhance generation accuracy, its effectiveness depends on the retrieval method and the number of retrieved chunks (k). BM25 underperforms the baseline, whereas semantic approaches (BERT and SPLADE) achieve better results, with SPLADE performing best at lower k values. However, excessive retrieval with high k parameter can lead to retrieving irrelevant chunks which degrades model performance. Our findings highlight the importance of optimizing retrieval configurations to balance context relevance and output consistency. This research provides insights into improving OCL rule generation using RAG and underscores the need for tailoring retrieval.

Figures (9)

• Figure 1: Example meta-model from b9
• Figure 2: Retrieval Augmented Generation Pipeline
• Figure 3: Comparison of Retriever Models
• Figure 4: Boxplot of Cosine Similarities BM25
• Figure 5: Boxplot of Euclidean Distances BM25