Eliciting Problem Specifications via Large Language Models
Robert E. Wray, James R. Kirk, John E. Laird
TL;DR
The paper investigates whether large language models can automatically translate natural-language problem descriptions into semi-formal problem-space specifications suitable for weak-method search in cognitive architectures. It proposes LLM-enabled cognitive task analysts (CTA) that leverage Newell's problem-space concepts and Polya's problem-solving heuristics to generate state definitions, operators, initial states, goal conditions, and search-control elements, all expressed in a form that can be consumed by downstream solvers. The approach is instantiated with a LangGraph-based multi-agent CTA prototype and evaluated in the water-jug domain, demonstrating that GPT-4 can produce precise, executable problem-space formulations and that refinement prompts improve quality and search efficiency. The work argues that automated problem specification can significantly accelerate cognitive-systems research by reducing human mediation, while identifying clear directions for extending the method to more problem classes, richer operator taxonomies, and declarative representations like PDDL. Formally, the problem is to find a solution path from an initial state $S_0$ to a goal state $S_g$ within a problem space $P = (S, O, C)$, where $S$ are states, $O$ are operators, and $C$ are constraints, and to do so with minimal search overhead through automated problem-space synthesis by LLMs.
Abstract
Cognitive systems generally require a human to translate a problem definition into some specification that the cognitive system can use to attempt to solve the problem or perform the task. In this paper, we illustrate that large language models (LLMs) can be utilized to map a problem class, defined in natural language, into a semi-formal specification that can then be utilized by an existing reasoning and learning system to solve instances from the problem class. We present the design of LLM-enabled cognitive task analyst agent(s). Implemented with LLM agents, this system produces a definition of problem spaces for tasks specified in natural language. LLM prompts are derived from the definition of problem spaces in the AI literature and general problem-solving strategies (Polya's How to Solve It). A cognitive system can then use the problem-space specification, applying domain-general problem solving strategies ("weak methods" such as search), to solve multiple instances of problems from the problem class. This result, while preliminary, suggests the potential for speeding cognitive systems research via disintermediation of problem formulation while also retaining core capabilities of cognitive systems, such as robust inference and online learning.