Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Procedural Adherence and Interpretability Through Neuro-Symbolic Generative Agents

Raven Rothkopf, Hannah Tongxin Zeng, Mark Santolucito

TL;DR

This paper addresses the difficulty of achieving long-horizon coherence and interpretability in LLM-based agents. It proposes a neuro-symbolic framework that combines Temporal Stream Logic (TSL) with reactive synthesis to synthesize an automaton that governs high-level prompt decisions, ensuring procedural adherence and enabling interpretability independent of the LLM internals. Empirical results on a choose-your-own-adventure task show automaton-enhanced agents achieving approximately 96–99% adherence to temporal constraints, while pure-LLM baselines lag and exhibit more hallucinations and arithmetic errors. The approach also provides modular, debuggable guarantees and a path toward scalable composition of agent behaviors.

Abstract

The surge in popularity of large language models (LLMs) has opened doors for new approaches to the creation of interactive agents. However, managing and interpreting the temporal behavior of such agents over the course of a potentially infinite interaction remain challenging. The stateful, long-term horizon reasoning required for coherent agent behavior does not fit well into the LLM paradigm. We propose a combination of formal logic-based program synthesis and LLM content generation to bring guarantees of procedural adherence and interpretability to generative agent behavior. To illustrate the benefit of procedural adherence and interpretability, we use Temporal Stream Logic (TSL) to generate an automaton that enforces an interpretable, high-level temporal structure on an agent. With the automaton tracking the context of the interaction and making decisions to guide the conversation accordingly, we can drive content generation in a way that allows the LLM to focus on a shorter context window. We evaluated our approach on different tasks involved in creating an interactive agent specialized for generating choose-your-own-adventure games. We found that over all of the tasks, an automaton-enhanced agent with procedural guarantees achieves at least 96% adherence to its temporal constraints, whereas a purely LLM-based agent demonstrates as low as 14.67% adherence.

Procedural Adherence and Interpretability Through Neuro-Symbolic Generative Agents

TL;DR

This paper addresses the difficulty of achieving long-horizon coherence and interpretability in LLM-based agents. It proposes a neuro-symbolic framework that combines Temporal Stream Logic (TSL) with reactive synthesis to synthesize an automaton that governs high-level prompt decisions, ensuring procedural adherence and enabling interpretability independent of the LLM internals. Empirical results on a choose-your-own-adventure task show automaton-enhanced agents achieving approximately 96–99% adherence to temporal constraints, while pure-LLM baselines lag and exhibit more hallucinations and arithmetic errors. The approach also provides modular, debuggable guarantees and a path toward scalable composition of agent behaviors.

Abstract

The surge in popularity of large language models (LLMs) has opened doors for new approaches to the creation of interactive agents. However, managing and interpreting the temporal behavior of such agents over the course of a potentially infinite interaction remain challenging. The stateful, long-term horizon reasoning required for coherent agent behavior does not fit well into the LLM paradigm. We propose a combination of formal logic-based program synthesis and LLM content generation to bring guarantees of procedural adherence and interpretability to generative agent behavior. To illustrate the benefit of procedural adherence and interpretability, we use Temporal Stream Logic (TSL) to generate an automaton that enforces an interpretable, high-level temporal structure on an agent. With the automaton tracking the context of the interaction and making decisions to guide the conversation accordingly, we can drive content generation in a way that allows the LLM to focus on a shorter context window. We evaluated our approach on different tasks involved in creating an interactive agent specialized for generating choose-your-own-adventure games. We found that over all of the tasks, an automaton-enhanced agent with procedural guarantees achieves at least 96% adherence to its temporal constraints, whereas a purely LLM-based agent demonstrates as low as 14.67% adherence.
Paper Structure (34 sections, 7 equations, 4 figures, 2 tables)

This paper contains 34 sections, 7 equations, 4 figures, 2 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. LLM Adherence and Interpretability
  4. Formal methods and LLMs
  5. Preliminaries
  6. Reactive Synthesis
  7. Generative Agents
  8. System Overview
  9. Generative Agent Correctness
  10. Adherence
  11. Point-in-time adherence
  12. Procedural adherence
  13. Interpretability
  14. Point-in-time interpretability
  15. Procedural interpretability
  16. ...and 19 more sections

Figures (4)

  • Figure 1: An example turn between an end-user and a choose-your-own-adventure game agent. After receiving a user prompt, the LLM consults an internal automaton synthesized from the Temporal Stream Logic (TSL) specification in Fig. \ref{['fig:adventure']}. Using previous story context, the automaton outputs a correct prompt modifier so that the agent's response aligns with the story's formal specification of long-horizon behavior. See Secs. \ref{['sec:agent']} and \ref{['sec:system']} for an in-depth explanation of the agent, Sec. \ref{['sec:correctness']} for formal descriptions of the agent's procedural adherence and interpretability, and Sec. \ref{['sec:eval']} for a preliminary agent evaluation.
  • Figure 2: A TSL spec (cf. Sec. \ref{['sec:synthesis']}) constraining the locations the player visits in a choose-your-own-adventure game.
  • Figure 3: Effect $\varphi_E$, trace $\pi$, cause automaton $\mathcal{A}_C$, and manually guessed cause $\varphi_C$ given the reactive system synthesized from Fig. \ref{['fig:adventure']}.
  • Figure 4: A snippet of the JavaScript code synthesized from the Fig \ref{['fig:adventure']} TSL spec representing a 5-state Mealy machine. The story summary has been abbreviated to s.