Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

The Impact of Critique on LLM-Based Model Generation from Natural Language: The Case of Activity Diagrams

Parham Khamsepour, Mark Cole, Ish Ashraf, DaYuan Tan, Sandeep Puri, Mehrdad Sabetzadeh, Shiva Nejati

TL;DR

This paper introduces LADEX, an LLM-driven critique-refine pipeline for generating activity diagrams from natural-language process descriptions, addressing structural correctness and semantic alignment. It presents five ablated variants to study the impact of the critique loop and of algorithmic versus LLM-based structural checks, along with two matching strategies (B-Match and L-Match) to evaluate correctness and completeness against ground-truth diagrams. Empirical results on a public PAGED dataset and an industry-Ciena dataset show that combining algorithmic structural checks with LLM-based alignment yields the best semantic quality (about 86% correctness and 92% completeness) with a modest number of LLM calls, while purely algorithmic structural checks offer a strong low-cost alternative. The findings support the value of iterative refinement for structural validity and semantic fidelity, and point to neuro-symbolic extensions as promising future work.

Abstract

Large Language Models (LLMs) show strong potential for automating model generation from natural-language descriptions. A common approach begins with an initial model generation, followed by an iterative critique-refine loop in which the model is evaluated for issues and refined based on those issues. This process needs to address: (1) structural correctness -- compliance with well-formedness rules -- and (2) semantic alignment -- accurate reflection of the intended meaning in the source text. We present LADEX (LLM-based Activity Diagram Extractor), a pipeline for deriving activity diagrams from natural-language process descriptions using an LLM-driven critique-refine process. Structural checks in LADEX can be performed either algorithmically or by an LLM, while alignment checks are performed by an LLM. We design five ablated variants of LADEX to study: (i) the impact of the critique-refine loop itself, (ii) the role of LLM-based semantic checks, and (iii) the comparative effectiveness of algorithmic versus LLM-based structural checks. To evaluate LADEX, we compare generated diagrams with expert ground truths using a trace-based behavioural and an LLM-based matcher. This enables automated measurement of correctness (whether the generated activity diagram includes the ground-truth nodes) and completeness (how many of the ground-truth nodes the generated activity diagram covers). Experiments on two datasets -- a public-domain dataset and an industry dataset from our collaborator, Ciena -- indicate: (1) Both matchers yield similar completeness and correctness comparisons. (2) The critique-refine loop improves structural validity, correctness, and completeness compared to single-pass generation. (3) Activity diagrams refined based on algorithmic structural checks achieve structural consistency, whereas those refined based on LLM-based checks often still show structural inconsistencies.

The Impact of Critique on LLM-Based Model Generation from Natural Language: The Case of Activity Diagrams

TL;DR

This paper introduces LADEX, an LLM-driven critique-refine pipeline for generating activity diagrams from natural-language process descriptions, addressing structural correctness and semantic alignment. It presents five ablated variants to study the impact of the critique loop and of algorithmic versus LLM-based structural checks, along with two matching strategies (B-Match and L-Match) to evaluate correctness and completeness against ground-truth diagrams. Empirical results on a public PAGED dataset and an industry-Ciena dataset show that combining algorithmic structural checks with LLM-based alignment yields the best semantic quality (about 86% correctness and 92% completeness) with a modest number of LLM calls, while purely algorithmic structural checks offer a strong low-cost alternative. The findings support the value of iterative refinement for structural validity and semantic fidelity, and point to neuro-symbolic extensions as promising future work.

Abstract

Large Language Models (LLMs) show strong potential for automating model generation from natural-language descriptions. A common approach begins with an initial model generation, followed by an iterative critique-refine loop in which the model is evaluated for issues and refined based on those issues. This process needs to address: (1) structural correctness -- compliance with well-formedness rules -- and (2) semantic alignment -- accurate reflection of the intended meaning in the source text. We present LADEX (LLM-based Activity Diagram Extractor), a pipeline for deriving activity diagrams from natural-language process descriptions using an LLM-driven critique-refine process. Structural checks in LADEX can be performed either algorithmically or by an LLM, while alignment checks are performed by an LLM. We design five ablated variants of LADEX to study: (i) the impact of the critique-refine loop itself, (ii) the role of LLM-based semantic checks, and (iii) the comparative effectiveness of algorithmic versus LLM-based structural checks. To evaluate LADEX, we compare generated diagrams with expert ground truths using a trace-based behavioural and an LLM-based matcher. This enables automated measurement of correctness (whether the generated activity diagram includes the ground-truth nodes) and completeness (how many of the ground-truth nodes the generated activity diagram covers). Experiments on two datasets -- a public-domain dataset and an industry dataset from our collaborator, Ciena -- indicate: (1) Both matchers yield similar completeness and correctness comparisons. (2) The critique-refine loop improves structural validity, correctness, and completeness compared to single-pass generation. (3) Activity diagrams refined based on algorithmic structural checks achieve structural consistency, whereas those refined based on LLM-based checks often still show structural inconsistencies.

Paper Structure

This paper contains 18 sections, 1 equation, 10 figures, 13 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Structural and Alignment Constraints
  3. Activity Diagram Generation and Refinement
  4. Activity Diagram Matching
  5. Behavioural Matching of Activity Diagrams (B-Match)
  6. LLM-Based Matching of Activity Diagrams (L-Match)
  7. Evaluation
  8. Datasets
  9. Evaluation Metrics
  10. Implementation
  11. Experimental Procedure
  12. Results
  13. Threats to Validity
  14. Related Work
  15. Conclusion
  16. ...and 3 more sections

Figures (10)

  • Figure 1: Motivating example
  • Figure 2: Example of one execution iteration of LADEX: (a) a candidate activity diagram generated from the process description in Figure \ref{['figure:example']}(a); (b) a critique of (a), evaluating the candidate against the structural and alignment constraints in Table \ref{['table:prompt_constraint']} and identifying any violations; (c) a refined version of (a), revised based on the critique in (b).
  • Figure 3: An overview of the architecture of LADEX.
  • Figure 4: A candidate activity diagram generated from the process description in Figure\ref{['figure:example']}(a) that is structurally sound based on the constraints outlined in Table\ref{['table:prompt_constraint']}, but is semantically misaligned with the textual process description.
  • Figure 5: Comparison of average correctness and completeness scores between L-Match (x-axis) and B-Match (y-axis): Blue and orange points, respectively, represent the average results of each variant across all runs and LLMs for the Paged and Industry datasets. Dashed lines indicate the best-fit linear trend for each dataset with Spearman correlation coefficients shown in the legend.
  • ...and 5 more figures

Theorems & Definitions (1)

  • Definition 2.1: Activity Diagram NejatiS0BM16