Black-box Context-free Grammar Inference for Readable & Natural Grammars

TL;DR

NatGI introduces a scalable, readable black-box CFG inference framework that integrates bracket-guided bubble exploration, LLM-guided bubble generation and non-terminal labeling, and hierarchical delta debugging to produce compact, semantically meaningful grammars. By extending TreeVada with these components and lexical inference, NatGI achieves higher F1 scores on large languages (e.g., lua, c, mysql) and yields grammars that align with human intuition. The approach demonstrates robust performance, beating strong baselines while maintaining reasonable runtimes, and delivers interpretable grammars suitable for debugging and validation. Overall, NatGI offers a practical solution for real-world grammar induction with improved accuracy and readability.

Abstract

Black-box context-free grammar inference is crucial for program analysis, reverse engineering, and security, yet existing tools such as Arvada, TreeVada, and Kedavra struggle with scalability, readability, and accuracy on large, complex languages. We present NatGI, a novel LLM-guided grammar inference framework that extends TreeVada's parse tree recovery with three key innovations: bracket-guided bubble exploration, LLM-driven bubble generation and non-terminal labeling, and hierarchical delta debugging (HDD) for systematic tree simplification. Bracket-guided exploration leverages syntactic cues such as parentheses to propose well-structured grammar fragments, while LLM guidance produces meaningful non-terminal names and selects more promising merges. Finally, HDD incrementally reduces unnecessary rules, which makes the grammars both compact and interpretable. In our experiments, we evaluate NatGI on a comprehensive benchmark suite ranging from small languages to larger ones such as lua, c, and mysql. Our results show that NatGI consistently outperforms strong baselines in terms of F1 score. On average, NatGI achieves an F1 score of 0.57, which is 25pp (percentage points) higher than the best-performing baseline, TreeVada. In the case of interpretability, our generated grammars perform significantly better than those produced by existing approaches. Leveraging LLM-based node renaming and bubble exploration, NatGI produces rules with meaningful non-terminal names and compact structures that align more closely with human intuition. As a result, developers and researchers can achieve higher accuracy while still being able to easily inspect, verify, and reason about the structure and semantics of the induced grammars.

Figures (10)

• Figure 1: Incremental parse tree construction: (1) parse tree segments at beginning, (2) node sequence $a==b$ bubbled-up under new node $t_{new}$, (3) $t_{new}$ and $true$ merged as $t_1$.
• Figure 2: Overview: After pre-processing NatGI iteratively generalizes its seed programs' parse trees (and thus grammar), first via brackets and LLM queries, then via TreeVada, before adding sub-trees via hierarchical delta debugging (HDD). NatGI generalizes its trees T with a candidate bubble b (detailed in right sidebar) if its black-box parser accepts all programs sampled from the tentatively generalized trees (b merged into T).
• Figure 3: Key NatGI processing steps on a toy seed program (top left): Tokenization and bracket-based tree pre-structuring (1), merging the new parent node $t_1$ with existing nodes & renaming $t_1$ via LLM suggestion (2), leveraging the bracketed sequence expr+expr as a successful bubble (3), merging LLM-suggested bubble c=expr with stmt (4), and pruning the tree via hierarchical delta debugging (5).
• Figure 4: LLM prompt for generating descriptive non-terminal labels.
• Figure 5: Prompt in generating bubble candidates. Full prompt available in the artifact.