Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

From a Natural to a Formal Language with DSL Assistant

My M. Mosthaf, Andrzej Wąsowski

TL;DR

The paper tackles the challenge of DSL development by integrating GPT-4o into a DSL workbench, DSL Assistant, which generates DSL grammars and example instances and supports non-linear co-evolution with automatic error repair. It presents an architecture based on a three-layer web application using Xtext, a PostgreSQL database, and an Eclipse server for grammar validation, with LLM prompts designed to produce JSON-style outputs. Through use-case scenarios and an extensive evaluation across multiple DSL domains, the study demonstrates that GPT-4o can produce syntactically valid grammars in about half of one-shot attempts and can repair most faulty grammars, while also highlighting variability based on domain and interaction patterns. The work shows practical potential for accelerating DSL design and iteration, while outlining limitations related to semantics, context, and multilingual applicability, and suggests directions for extending coverage to MDSE semantics and alternative LLMs.

Abstract

The development of domain-specific languages (DSLs) is a laborious and iterative process that seems to naturally lean to the use of generative artificial intelligence. We design and prototype DSL Assistant, a tool that integrates generative language models to support the development of DSLs. DSL Assistant uses OpenAI's assistant API with GPT-4o to generate DSL grammars and example instances. To reflect real-world use, DSL Assistant supports several different interaction modes for evolving a DSL design, and includes automatic error repair. Our experiments show that DSL Assistant helps users to create and modify DSLs. However, the quality of the generated DSLs depends on the specific domain and the followed interaction patterns.

From a Natural to a Formal Language with DSL Assistant

TL;DR

The paper tackles the challenge of DSL development by integrating GPT-4o into a DSL workbench, DSL Assistant, which generates DSL grammars and example instances and supports non-linear co-evolution with automatic error repair. It presents an architecture based on a three-layer web application using Xtext, a PostgreSQL database, and an Eclipse server for grammar validation, with LLM prompts designed to produce JSON-style outputs. Through use-case scenarios and an extensive evaluation across multiple DSL domains, the study demonstrates that GPT-4o can produce syntactically valid grammars in about half of one-shot attempts and can repair most faulty grammars, while also highlighting variability based on domain and interaction patterns. The work shows practical potential for accelerating DSL design and iteration, while outlining limitations related to semantics, context, and multilingual applicability, and suggests directions for extending coverage to MDSE semantics and alternative LLMs.

Abstract

The development of domain-specific languages (DSLs) is a laborious and iterative process that seems to naturally lean to the use of generative artificial intelligence. We design and prototype DSL Assistant, a tool that integrates generative language models to support the development of DSLs. DSL Assistant uses OpenAI's assistant API with GPT-4o to generate DSL grammars and example instances. To reflect real-world use, DSL Assistant supports several different interaction modes for evolving a DSL design, and includes automatic error repair. Our experiments show that DSL Assistant helps users to create and modify DSLs. However, the quality of the generated DSLs depends on the specific domain and the followed interaction patterns.
Paper Structure (7 sections, 7 figures, 4 tables)

This paper contains 7 sections, 7 figures, 4 tables.

Table of Contents

  1. Introduction
  2. Background
  3. Analysis and Requirements
  4. The Implementation of DSL Assistant
  5. Evaluation Design
  6. Results
  7. Conclusion

Figures (7)

  • Figure 1: The architecture of DSL Assistant.
  • Figure 2: LLM interaction contexts as a feature model
  • Figure 3: A fragment of the DSL Assistant GUI with the current grammar and meta-model
  • Figure 4: State machine diagram over the version processing in DSL Assistant
  • Figure 5: Example prompts used by DSL Assistant. Left: A prompt for a DSL kind, properties input, and no base. Right: A prompt constructed for a DSL kind with an error message input and a faulty DSL version as a base
  • ...and 2 more figures