David vs. Goliath: A comparative study of different-sized LLMs for code generation in the domain of automotive scenario generation

TL;DR

This work addresses the reproducibility and comparability gap in NL-to-Scenic code generation for automotive scenario simulation by introducing NL2Scenic, an open dataset of 146 NL–Scenic pairs with a 30-case difficulty-stratified test split, plus an Example Retriever and 14 prompting strategies. It conducts a comprehensive, model-wide evaluation across 13 LLMs (proprietary and open-source) using both text-based and execution-based metrics, and validates them against expert judgments, proposing $EDIT$-$SIM$ as the best single metric and $EDIT$-$COMP$ as a robust dataset-level proxy. Results show GPT-4o is the strongest performer overall, while mid-size open-source models (via FSER prompts) can approach state-of-the-art performance at a lower cost, with scaling yielding diminishing returns beyond mid-size. The framework and metrics offer a standardized, reproducible basis for evaluating Scenic code generation and suggest that practical, cost-effective open-source LLMs can support autonomous-driving scenario programming at scale.

Abstract

Scenario simulation is central to testing autonomous driving systems. Scenic, a domain-specific language (DSL) for CARLA, enables precise and reproducible scenarios, but NL-to-Scenic generation with large language models (LLMs) suffers from scarce data, limited reproducibility, and inconsistent metrics. We introduce NL2Scenic, an open dataset and framework with 146 NL/Scenic pairs, a difficulty-stratified 30-case test split, an Example Retriever, and 14 prompting variants (ZS, FS, CoT, SP, MoT). We evaluate 13 models: four proprietary (GPT-4o, GPT-5, Claude-Sonnet-4, Gemini-2.5-pro) and nine open-source code models (Qwen2.5Coder 0.5B-32B; CodeLlama 7B/13B/34B), using text metrics (BLEU, ChrF, EDIT-SIM, CrystalBLEU) and execution metrics (compilation and generation), and compare them with an expert study (n=11). EDIT-SIM correlates best with human judgments; we also propose EDIT-COMP (F1 of EDIT-SIM and compilation) as a robust dataset-level proxy that improves ranking fidelity. GPT-4o performs best overall, while Qwen2.5Coder-14B reaches about 88 percent of its expert score on local hardware. Retrieval-augmented prompting, Few-Shot with Example Retriever (FSER), consistently boosts smaller models, and scaling shows diminishing returns beyond mid-size, with Qwen2.5Coder outperforming CodeLlama at comparable scales. NL2Scenic and EDIT-COMP offer a standardized, reproducible basis for evaluating Scenic code generation and indicate that mid-size open-source models are practical, cost-effective options for autonomous-driving scenario programming.

Figures (10)

• Figure 1: Architecture of NL2Scenic framework. The system takes a NL scenario description (.txt) as input and generates executable Scenic code (.scenic) through three main components: (1) the Prompt Generator, which creates tailored prompts; (2) the Example Retriever, which retrieves similar examples from a database; and (3) the Generation Engine, which synthesizes the prompt components and invokes the LLM for code generation.
• Figure 2: Scenic spatial operators for defining object relationships, including directional positioning (e.g., left of, back right of), point-based offsets, and relative coordinates. Adapted from ScenicBookChapter.
• Figure 3: Top: four CARLA simulations generated from single Scenic script. Bottom: corresponding Scenic script.
• Figure 4: Example CARLA renderings from \githublink Scenic scripts. The figure shows two dinamic scenarios of the dataset. In the first scenario the ego yields to an oncoming car. The second scenario shows a multi vehicle scenario taking place on a multilane road.
• Figure 5: Overview of the Example Retriever (left) and the Prompt Generator (right).