Specifications: The missing link to making the development of LLM systems an engineering discipline

TL;DR

This paper argues that making LLM-based systems an engineering discipline hinges on precise task and outcome specifications, separating statement specifications (what to do) from solution specifications (how to verify it). It introduces five core software-engineering properties—verifiability, debuggability, modularity, reusability, and automated decision making—and applies them to LLM contexts, outlining challenges and concrete approaches (structured outputs, process supervision, testing, and modular architectures). The authors survey the ambiguity inherent in prompts, propose disambiguation techniques inspired by human communication, and discuss how to build reliable, scalable, and autonomous AI systems through improved specifications. The work aims to accelerate the development of modular, reliable LLM ecosystems with practical impact across AI-powered software and services by providing a formal framework and research directions for specification-driven engineering.

Abstract

Despite the significant strides made by generative AI in just a few short years, its future progress is constrained by the challenge of building modular and robust systems. This capability has been a cornerstone of past technological revolutions, which relied on combining components to create increasingly sophisticated and reliable systems. Cars, airplanes, computers, and software consist of components-such as engines, wheels, CPUs, and libraries-that can be assembled, debugged, and replaced. A key tool for building such reliable and modular systems is specification: the precise description of the expected behavior, inputs, and outputs of each component. However, the generality of LLMs and the inherent ambiguity of natural language make defining specifications for LLM-based components (e.g., agents) both a challenging and urgent problem. In this paper, we discuss the progress the field has made so far-through advances like structured outputs, process supervision, and test-time compute-and outline several future directions for research to enable the development of modular and reliable LLM-based systems through improved specifications.

Figures (12)

• Figure 1: A car consists of many parts (modularity), where some parts are shared with other cars (reusability). When something goes wrong, the mechanic diagnoses the problem (verifiability) and then fixes it (debuggability). Furthermore, some functions of the car don’t require human intervention (automated decision making). To enable these properties we need detailed schematics and detailed descriptions of how each of these components works (specifications).
• Figure 2: (a) A task specification describes (i) what the task should do (i.e., statement specification) and (ii) how should the task's solution be verified (i.e., solution specification). (b) Some tasks, such as "Will AGI happen by 2027?" are ambiguous both in terms of statement and solution specifications (e.g., AGI is not well defined, and there is no clear way to evaluate how good an answer (solution) is.), while some tasks such as "Solve $x^3 - x = 0$" are unambiguous both in terms of statement and solution specifications (e.g., one can easily verify the correctness of the solution by substituting it in the equation).
• Figure 3: A simple example of prompt (statement) disambiguation. (a) The user wants to learn how long it takes to drive from Venice, CA to Paris, TX, but doesn’t mention all details in her question. As a result, the agent gives the wrong answer, i.e., the flight time from Venice, Italy to Paris, France. Two simple methods to disambiguate: (b) the agent asks for clarification about which cities and the method of transportation; (c) the agent states all the assumptions in its answer, so the user sees how to improve the prompt.
• Figure 4: The main system components of a SQL query engine.
• Figure 5: High-level architecture of (a) developing a program for a given task vs (b) using an LLM to accomplish the same task.The LLM obviates the need for writing the program and doesn’t need a detailed specification. The LLM prompt contains both the description (specification) of the task and its input.