Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

DocCGen: Document-based Controlled Code Generation

Sameer Pimparkhede, Mehant Kammakomati, Srikanth Tamilselvam, Prince Kumar, Ashok Pon Kumar, Pushpak Bhattacharyya

TL;DR

DocCGen addresses NL-to-Code for structured DSLs by leveraging enterprise library documentation to constrain generation. It decomposes the task into library detection via information retrieval and constrained decoding using grammar and templates extracted from docs, improving correctness on unseen libraries. It contributes a publicly available Ansible YAML NL-to-code dataset and augments existing Bash datasets, with consistent improvements across OOD and ID settings for multiple models. This approach reduces syntactic and semantic errors in constrained DSL generation, enabling practical NL-to-DSL tooling.

Abstract

Recent developments show that Large Language Models (LLMs) produce state-of-the-art performance on natural language (NL) to code generation for resource-rich general-purpose languages like C++, Java, and Python. However, their practical usage for structured domain-specific languages (DSLs) such as YAML, JSON is limited due to domain-specific schema, grammar, and customizations generally unseen by LLMs during pre-training. Efforts have been made to mitigate this challenge via in-context learning through relevant examples or by fine-tuning. However, it suffers from problems, such as limited DSL samples and prompt sensitivity but enterprises maintain good documentation of the DSLs. Therefore, we propose DocCGen, a framework that can leverage such rich knowledge by breaking the NL-to-Code generation task for structured code languages into a two-step process. First, it detects the correct libraries using the library documentation that best matches the NL query. Then, it utilizes schema rules extracted from the documentation of these libraries to constrain the decoding. We evaluate our framework for two complex structured languages, Ansible YAML and Bash command, consisting of two settings: Out-of-domain (OOD) and In-domain (ID). Our extensive experiments show that DocCGen consistently improves different-sized language models across all six evaluation metrics, reducing syntactic and semantic errors in structured code. We plan to open-source the datasets and code to motivate research in constrained code generation.

DocCGen: Document-based Controlled Code Generation

TL;DR

DocCGen addresses NL-to-Code for structured DSLs by leveraging enterprise library documentation to constrain generation. It decomposes the task into library detection via information retrieval and constrained decoding using grammar and templates extracted from docs, improving correctness on unseen libraries. It contributes a publicly available Ansible YAML NL-to-code dataset and augments existing Bash datasets, with consistent improvements across OOD and ID settings for multiple models. This approach reduces syntactic and semantic errors in constrained DSL generation, enabling practical NL-to-DSL tooling.

Abstract

Recent developments show that Large Language Models (LLMs) produce state-of-the-art performance on natural language (NL) to code generation for resource-rich general-purpose languages like C++, Java, and Python. However, their practical usage for structured domain-specific languages (DSLs) such as YAML, JSON is limited due to domain-specific schema, grammar, and customizations generally unseen by LLMs during pre-training. Efforts have been made to mitigate this challenge via in-context learning through relevant examples or by fine-tuning. However, it suffers from problems, such as limited DSL samples and prompt sensitivity but enterprises maintain good documentation of the DSLs. Therefore, we propose DocCGen, a framework that can leverage such rich knowledge by breaking the NL-to-Code generation task for structured code languages into a two-step process. First, it detects the correct libraries using the library documentation that best matches the NL query. Then, it utilizes schema rules extracted from the documentation of these libraries to constrain the decoding. We evaluate our framework for two complex structured languages, Ansible YAML and Bash command, consisting of two settings: Out-of-domain (OOD) and In-domain (ID). Our extensive experiments show that DocCGen consistently improves different-sized language models across all six evaluation metrics, reducing syntactic and semantic errors in structured code. We plan to open-source the datasets and code to motivate research in constrained code generation.
Paper Structure (58 sections, 4 figures, 11 tables)

This paper contains 58 sections, 4 figures, 11 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. DocCGen Framework
  4. Background and Definitions
  5. Structured schema:
  6. Template:
  7. Trigger signals:
  8. Framework
  9. Information retrieval
  10. Sparse retrieval
  11. Dense retrieval
  12. Constrained generation
  13. String Selection:
  14. Library selection:
  15. Generating variable part:
  16. ...and 43 more sections

Figures (4)

  • Figure 1: Illustration of shortcomings with fine-tuning and DocPrompting zhou2022doccoder approaches with an example for (a) NL to Bash task (uses GPT Neo 1.3B) and (b) NL to Ansible-YAML task (uses StarCoder2 3B) and the proposed DocCGen method to overcome the limitations.
  • Figure 2: Overview of DocCGen. For a given user query, top $k$ relevant library documentations are retrieved and for which initial $k$ templates are created. Static part of the template is shown in red, while the variable part is in blue. The variable field with a fixed position in the code is enclosed in angle brackets, for instance <subcommand>, as shown in the initial k templates block in the figure. The model is guided to follow one of the templates during decoding. Each time step $t_i$ shows the step-by-step dynamic template evolution and constrained decoding output, adhering to the time-step template leading to the final generated code at $t3$.
  • Figure 3: Demonstration of the performance of StarCoder 1B for NL to Ansible-YAML task over varying number of train samples per module for in domain setting.
  • Figure 4: Demonstration of the performance of (a) (b) (c) GPT Neo 1.3B, (d) (e) (f) StarCoder2 3B, and (g) (h) (i) StarCoder2 7B in different configurations for NL to Ansible-YAML task over varying number of train samples per module for in domain setting. We omit CodeLlama 34B as it is evaluated in few-shot setting.