Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Anka: A Domain-Specific Language for Reliable LLM Code Generation

Saif Khalfan Saif Al Mazrouei

TL;DR

This work introduces Anka, a domain-specific language for data transformation pipelines designed with constrained syntax to reduce LLM-generated errors. Through zero-shot in-context learning, Claude 3.5 Haiku achieves near-native parsing (99.9%) and attains a 40-point advantage over Python on multi-step tasks, with cross-model validation by GPT-4o-mini confirming the improvement. The key contributions include a 100-task benchmark, analysis of error patterns, and design principles promoting reliable LLM code generation via canonical forms, explicit state naming, and step-wise scaffolding. The results suggest that purpose-built DSLs tailored to LLMs can outperform general-purpose languages, offering a promising direction for improving reliability in automated code generation and prompting further exploration of DSL design for AI-assisted programming.

Abstract

Large Language Models (LLMs) have demonstrated remarkable capabilities in code generation, yet they exhibit systematic errors on complex, multi-step programming tasks. We hypothesize that these errors stem from the flexibility of general-purpose languages, which permits multiple valid approaches and requires implicit state management. To test this hypothesis, we introduce Anka, a domain-specific language (DSL) for data transformation pipelines designed with explicit, constrained syntax that reduces ambiguity in code generation. Despite having zero prior training exposure to Anka, Claude 3.5 Haiku achieves 99.9% parse success and 95.8% overall task accuracy across 100 benchmark problems. Critically, Anka demonstrates a 40 percentage point accuracy advantage over Python on multi-step pipeline tasks (100% vs. 60%), where Python's flexible syntax leads to frequent errors in operation sequencing and variable management. Cross-model validation with GPT-4o-mini confirms this advantage (+26.7 percentage points on multi-step tasks). Our results demonstrate that: (1) LLMs can learn novel DSLs entirely from in-context prompts, achieving near-native accuracy; (2) constrained syntax significantly reduces errors on complex tasks; and (3) domain-specific languages purposefully designed for LLM generation can outperform general-purpose languages on which the LLM has extensive training. We release the complete language implementation, benchmark suite, and evaluation framework to facilitate further research.

Anka: A Domain-Specific Language for Reliable LLM Code Generation

TL;DR

This work introduces Anka, a domain-specific language for data transformation pipelines designed with constrained syntax to reduce LLM-generated errors. Through zero-shot in-context learning, Claude 3.5 Haiku achieves near-native parsing (99.9%) and attains a 40-point advantage over Python on multi-step tasks, with cross-model validation by GPT-4o-mini confirming the improvement. The key contributions include a 100-task benchmark, analysis of error patterns, and design principles promoting reliable LLM code generation via canonical forms, explicit state naming, and step-wise scaffolding. The results suggest that purpose-built DSLs tailored to LLMs can outperform general-purpose languages, offering a promising direction for improving reliability in automated code generation and prompting further exploration of DSL design for AI-assisted programming.

Abstract

Large Language Models (LLMs) have demonstrated remarkable capabilities in code generation, yet they exhibit systematic errors on complex, multi-step programming tasks. We hypothesize that these errors stem from the flexibility of general-purpose languages, which permits multiple valid approaches and requires implicit state management. To test this hypothesis, we introduce Anka, a domain-specific language (DSL) for data transformation pipelines designed with explicit, constrained syntax that reduces ambiguity in code generation. Despite having zero prior training exposure to Anka, Claude 3.5 Haiku achieves 99.9% parse success and 95.8% overall task accuracy across 100 benchmark problems. Critically, Anka demonstrates a 40 percentage point accuracy advantage over Python on multi-step pipeline tasks (100% vs. 60%), where Python's flexible syntax leads to frequent errors in operation sequencing and variable management. Cross-model validation with GPT-4o-mini confirms this advantage (+26.7 percentage points on multi-step tasks). Our results demonstrate that: (1) LLMs can learn novel DSLs entirely from in-context prompts, achieving near-native accuracy; (2) constrained syntax significantly reduces errors on complex tasks; and (3) domain-specific languages purposefully designed for LLM generation can outperform general-purpose languages on which the LLM has extensive training. We release the complete language implementation, benchmark suite, and evaluation framework to facilitate further research.
Paper Structure (54 sections, 1 figure, 4 tables)

This paper contains 54 sections, 1 figure, 4 tables.

Table of Contents

  1. Introduction
  2. Contributions.
  3. Related Work
  4. LLM Code Generation.
  5. Domain-Specific Languages.
  6. Prompt Engineering and Constrained Generation.
  7. The Anka Language
  8. Design Principles
  9. Principle 1: One Canonical Form.
  10. Principle 2: Named Intermediate Results.
  11. Principle 3: Explicit Step Structure.
  12. Principle 4: Verbose Keywords.
  13. Syntax Overview
  14. Type Declarations.
  15. Operations.
  16. ...and 39 more sections

Figures (1)

  • Figure 1: Anka advantage grows with task complexity. Simple tasks (1--2 operations) show no advantage; complex tasks (5+ operations) show +40% advantage.