The Pneuma Project: Reifying Information Needs as Relational Schemas to Automate Discovery, Guide Preparation, and Align Data with Intent

TL;DR

The paper tackles the data discovery and preparation bottleneck when user intent is vague by reifying information needs as a relational schema $(T,Q)$ and guiding iterative, language-driven discovery toward a usable document. It introduces Pneuma-Seeker, a system built on context specialization, a conductor-style planner, and a convergence mechanism that treats $(T,Q)$ as a shared state between user and machine. Through LLM-based simulations on benchmark datasets, Pneuma-Seeker demonstrates improved convergence and accuracy over static or retrieval-only baselines, while also surfacing intermediate states to capture tacit organizational knowledge. The approach enables semi-automatic, language-guided workflows and positions Pneuma-Seeker as a foundation for emergent documentation and potential internal data markets within organizations.

Abstract

Data discovery and preparation remain persistent bottlenecks in the data management lifecycle, especially when user intent is vague, evolving, or difficult to operationalize. The Pneuma Project introduces Pneuma-Seeker, a system that helps users articulate and fulfill information needs through iterative interaction with a language model-powered platform. The system reifies the user's evolving information need as a relational data model and incrementally converges toward a usable document aligned with that intent. To achieve this, the system combines three architectural ideas: context specialization to reduce LLM burden across subtasks, a conductor-style planner to assemble dynamic execution plans, and a convergence mechanism based on shared state. The system integrates recent advances in retrieval-augmented generation (RAG), agentic frameworks, and structured data preparation to support semi-automatic, language-guided workflows. We evaluate the system through LLM-based user simulations and show that it helps surface latent intent, guide discovery, and produce fit-for-purpose documents. It also acts as an emergent documentation layer, capturing institutional knowledge and supporting organizational memory.

Figures (5)

• Figure 1: The Architecture of Pneuma-Seeker
• Figure 2: Interface of Pneuma-Seeker, showing: [1] User Query (Clarification), [2] User-Facing Message, and [3] State View Page $(T,Q)$. Note: the numbers and values of $T$ shown here are not real for privacy reasons.
• Figure 3: Prompt for LLM_Sim
• Figure 4: Comparison of Median Turns to Convergence vs. Convergence Percentage (Archeology Dataset)
• Figure 5: Comparison of Median Turns to Convergence vs. Convergence Percentage (Environment Dataset)