Using Grammar Masking to Ensure Syntactic Validity in LLM-based Modeling Tasks

TL;DR

This work tackles the problem of generating syntactically valid domain-specific language artifacts with large language models, where traditional prompt-based methods struggle with complex grammars. It introduces grammar masking, a constrained-decoding approach that ties LLM output to a target CFG through Guidance-AI, a parser-guided generation framework that uses a Trie for tokens and an Earley parser to enforce syntactic validity. The method is instantiated on MontiCore-based DSLs (Structured English SEN and CD4A) and evaluated against a few-shot baseline, showing substantial improvements in parseable outputs across multiple LLMs (e.g., CD4A: 41.97% to 92.63% parsable for Llama3 8B 4-Bit; SEN: 26.54% to 90.26% parsable). While grammar masking markedly increases syntactic correctness, it incurs higher runtime costs and does not address semantic validation; the authors argue it is particularly beneficial when prompt engineering alone cannot achieve satisfactory results and when offline, smaller models are required. Future work includes precomputation optimizations and extending the approach to additional grammars and context conditions.

Abstract

We present and evaluate a method called grammar masking, which is used to guide large language models (LLMs) toward producing syntactically correct models for a given context-free grammar. Prompt engineering methods such as few-shot learning or priming can be used to improve the chances of an LLM producing correct syntax, but the more complex the grammar, the more time-consuming and less promising these methods become. Previous work is focused primarily on the usage of either language model training or prompt engineering. In this work, a method is presented that restricts the output to a given grammar using constrained decoding to ensure the output adheres to a valid syntax. We use several DSLs built with MontiCore and task multiple LLMs to produce models with and without constrained decoding. A corresponding parser is used to confirm the syntactic correctness of each model. We show that grammar masking can dramatically improve the modeling capabilities of several LLMs, reducing the need for well-refined prompting while increasing the chance of producing correct models.

Figures (4)

• Figure 1: Grammar Tree Structure
• Figure 2: Evaluating the performance of a few-shot learning-based approach
• Figure 3: Combining the Guidance Framework with MontiCore to generate syntactically valid models.
• Figure 4: Evaluating the performance of a grammar-masking-based approach