Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Detecting Semantic Alignments between Textual Specifications and Domain Models

Shwetali Shimangaud, Lola Burgueño, Rijul Saini, Jörg Kienzle

TL;DR

This work proposes an approach for determining the alignment between a partial domain model and a textual specification that is integrated into a modeling tool to provide positive feedback or generate warnings, or employed for offline validation and quality assessment.

Abstract

Context: Having domain models derived from textual specifications has proven to be very useful in the early phases of software engineering. However, creating correct domain models and establishing clear links with the textual specification is a challenging task, especially for novice modelers. Objectives: We propose an approach for determining the alignment between a partial domain model and a textual specification. Methods: To this aim, we use Natural Language Processing techniques to pre-process the text, generate an artificial natural language specification for each model element, and then use an LLM to compare the generated description with matched sentences from the original specification. Ultimately, our algorithm classifies each model element as either aligned (i.e., correct), misaligned (i.e., incorrect), or unclassified (i.e., insufficient evidence). Furthermore, it outputs the related sentences from the textual specification that provide the evidence for the determined class. Results: We have evaluated our approach on a set of examples from the literature containing diverse domains, each consisting of a textual specification and a reference domain model, as well as on models containing modeling errors that were systematically derived from the correct models through mutation. Our results show that we are able to identify alignments and misalignments with a precision close to 1 and a recall of approximately 78%, with execution times ranging from 18 seconds to 1 minute per model element. Conclusion: Since our algorithm almost never classifies model elements incorrectly, and is able to classify over 3/4 of the model elements, it could be integrated into a modeling tool to provide positive feedback or generate warnings, or employed for offline validation and quality assessment.

Detecting Semantic Alignments between Textual Specifications and Domain Models

TL;DR

This work proposes an approach for determining the alignment between a partial domain model and a textual specification that is integrated into a modeling tool to provide positive feedback or generate warnings, or employed for offline validation and quality assessment.

Abstract

Context: Having domain models derived from textual specifications has proven to be very useful in the early phases of software engineering. However, creating correct domain models and establishing clear links with the textual specification is a challenging task, especially for novice modelers. Objectives: We propose an approach for determining the alignment between a partial domain model and a textual specification. Methods: To this aim, we use Natural Language Processing techniques to pre-process the text, generate an artificial natural language specification for each model element, and then use an LLM to compare the generated description with matched sentences from the original specification. Ultimately, our algorithm classifies each model element as either aligned (i.e., correct), misaligned (i.e., incorrect), or unclassified (i.e., insufficient evidence). Furthermore, it outputs the related sentences from the textual specification that provide the evidence for the determined class. Results: We have evaluated our approach on a set of examples from the literature containing diverse domains, each consisting of a textual specification and a reference domain model, as well as on models containing modeling errors that were systematically derived from the correct models through mutation. Our results show that we are able to identify alignments and misalignments with a precision close to 1 and a recall of approximately 78%, with execution times ranging from 18 seconds to 1 minute per model element. Conclusion: Since our algorithm almost never classifies model elements incorrectly, and is able to classify over 3/4 of the model elements, it could be integrated into a modeling tool to provide positive feedback or generate warnings, or employed for offline validation and quality assessment.
Paper Structure (50 sections, 11 equations, 4 figures, 4 tables, 1 algorithm)

This paper contains 50 sections, 11 equations, 4 figures, 4 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Motivation and Running Example
  3. Approach
  4. Overview
  5. NLP Specification Preprocessor
  6. Model Slicer
  7. Semantic Matcher
  8. Sentence Generator
  9. Attributes
  10. Association (and Aggregation) Relationships
  11. Composition Relationships
  12. Inheritance Relationships
  13. Enumerations
  14. LLM-based Semantic (Mis)Alignment Detection
  15. Prompt Engineering
  16. ...and 35 more sections

Figures (4)

  • Figure 1: Running Example: Car Service Specification
  • Figure 2: Running Example: Car Service Domain Model
  • Figure 3: Approach Overview
  • Figure 4: Classification Workflow