Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

LLMs Can't Play Hangman: On the Necessity of a Private Working Memory for Language Agents

Davide Baldelli, Ali Parviz, Amal Zouaq, Sarath Chandar

TL;DR

This work investigates whether LLM-based agents can reliably maintain hidden state when interacting via a standard chat interface. It defines Private State Interactive Tasks (PSITs) and proves that public-only agents cannot guarantee secrecy and consistency in PSITs, using Hangman and a Diagnosis Simulator as canonical examples. The authors propose a private working memory architecture and two implementation schemes (workflow and autonomous) to restore state persistence, supported by a self-consistency testing protocol with a forking evaluation. Empirical results show that memory-augmented agents with private working memory achieve near-perfect self-consistency, while retrieval-based baselines fail to preserve hidden state, and scalability of the private memory improves performance. The findings argue that private working memory is a necessary component for reliable interactive language agents and guide future memory-augmented system design.

Abstract

As LLMs move from text completion toward autonomous agents, they remain constrained by the standard chat interface, which lacks private working memory. This raises a fundamental question: can agents reliably perform interactive tasks that depend on hidden state? We define Private State Interactive Tasks (PSITs), which require agents to generate and maintain hidden information while producing consistent public responses. We show theoretically that any agent restricted to the public conversation history cannot simultaneously preserve secrecy and consistency in PSITs, yielding an impossibility theorem. To empirically validate this limitation, we introduce a self-consistency testing protocol that evaluates whether agents can maintain a hidden secret across forked dialogue branches. Standard chat-based LLMs and retrieval-based memory baselines fail this test regardless of scale, demonstrating that semantic retrieval does not enable true state maintenance. To address this, we propose a novel architecture incorporating an explicit private working memory; we demonstrate that this mechanism restores consistency, establishing private state as a necessary component for interactive language agents.

LLMs Can't Play Hangman: On the Necessity of a Private Working Memory for Language Agents

TL;DR

This work investigates whether LLM-based agents can reliably maintain hidden state when interacting via a standard chat interface. It defines Private State Interactive Tasks (PSITs) and proves that public-only agents cannot guarantee secrecy and consistency in PSITs, using Hangman and a Diagnosis Simulator as canonical examples. The authors propose a private working memory architecture and two implementation schemes (workflow and autonomous) to restore state persistence, supported by a self-consistency testing protocol with a forking evaluation. Empirical results show that memory-augmented agents with private working memory achieve near-perfect self-consistency, while retrieval-based baselines fail to preserve hidden state, and scalability of the private memory improves performance. The findings argue that private working memory is a necessary component for reliable interactive language agents and guide future memory-augmented system design.

Abstract

As LLMs move from text completion toward autonomous agents, they remain constrained by the standard chat interface, which lacks private working memory. This raises a fundamental question: can agents reliably perform interactive tasks that depend on hidden state? We define Private State Interactive Tasks (PSITs), which require agents to generate and maintain hidden information while producing consistent public responses. We show theoretically that any agent restricted to the public conversation history cannot simultaneously preserve secrecy and consistency in PSITs, yielding an impossibility theorem. To empirically validate this limitation, we introduce a self-consistency testing protocol that evaluates whether agents can maintain a hidden secret across forked dialogue branches. Standard chat-based LLMs and retrieval-based memory baselines fail this test regardless of scale, demonstrating that semantic retrieval does not enable true state maintenance. To address this, we propose a novel architecture incorporating an explicit private working memory; we demonstrate that this mechanism restores consistency, establishing private state as a necessary component for interactive language agents.
Paper Structure (36 sections, 2 theorems, 5 equations, 6 figures, 1 table)

This paper contains 36 sections, 2 theorems, 5 equations, 6 figures, 1 table.

Table of Contents

  1. Introduction
  2. Contributions.
  3. Related Works
  4. Agentic Frameworks.
  5. External Memory Systems.
  6. Cognitive Architectures.
  7. Game-Based and Memory Benchmarks.
  8. Faithfulness and Self-Consistency.
  9. Private State Interactive Tasks (PSITs)
  10. Implications for Existing Agents.
  11. Self-Consistency Testing for PSITs
  12. Hangman.
  13. Diagnosis Simulator.
  14. Evaluation.
  15. Comparative Baselines.
  16. ...and 21 more sections

Key Result

Theorem 1

No POCA can simultaneously guarantee both secrecy and consistency in a PSIT.

Figures (6)

  • Figure 1: The Self-Consistency Testing Protocol and Memory-Augmented Architecture.a and b illustrate the forking mechanism in Hangman and Diagnosis Simulator, where interaction branches test whether the agent maintains a single hidden secret. Success (green checks) requires confirming the true secret while rejecting valid alternatives; failure (red crosses) occurs when multiple candidates are affirmed or the true secret is denied. c shows a memory-augmented agent that maintains a private working memory updated each turn and reinjected into the prompt, enabling consistent preservation of hidden state. Prompts are shortened for readability; full versions appear in the Appendix. \ref{['app:prompts']}.
  • Figure 2: Agent architectures. Autonomous agents delegate branching decisions to the LLM, which chooses actions dynamically, while workflows follow a fixed, pre-defined sequence of calls.
  • Figure 3: Visualizing the Impossibility Theorem. The public history $H_{t-1}$ is compatible with two distinct secrets ($s, s'$). Given an input $x_t$, the task rules $\mathcal{R}$ require distinct outputs $y$ and $y'$. However, a Public-Only Chat Agent (POCA) conditions only on public information ($H_{t-1}, x_t$), producing a single distribution $\pi$ that cannot simultaneously match both deterministic targets.
  • Figure 4: Failure Mode Analysis. Distribution of interaction outcomes across 50 runs per condition. Self-Consistent (Green) indicates the agent maintained secrecy and affirmed only its hidden secret. Leakage (Blue) denotes runs where the secret was explicitly revealed in the public transcript. Over-Confirmation (Yellow) indicates the agent maintained secrecy but affirmed multiple conflicting candidates. State Substitution (Orange) and All Denial (Red) indicate runs where the agent maintained secrecy but failed to correctly affirm its revealed secret .
  • Figure 5: Efficiency Analysis. Evolution of private state size (in tokens) over dialogue turns on a logarithmic scale. Shaded regions indicate the 95% confidence interval. Private CoT (Grey) requires approximately 10× more context than explicit memory methods.
  • ...and 1 more figures

Theorems & Definitions (7)

  • Definition 1: Public-Only Chat Agent (POCA)
  • Definition 2: Private State Interactive Tasks (PSITs)
  • Definition 3: Consistency
  • Definition 4: Secrecy
  • Theorem 1: Impossibility of Secrecy and Consistency for POCAs
  • Lemma 1: Non-implementability of PSIT under public-only chat
  • proof