Steering Semantic Data Processing With DocWrangler
Shreya Shankar, Bhavya Chopra, Mawil Hasan, Stephen Lee, Björn Hartmann, Joseph M. Hellerstein, Aditya G. Parameswaran, Eugene Wu
TL;DR
DocWrangler tackles the challenge of scalable semantic data processing on unstructured text by introducing a mixed-initiative IDE that treats LLMs as semantic operators. Its three core innovations—in-situ user notes, LLM-assisted prompt refinement, and LLM-assisted operation decomposition—address the gulfs of comprehension, specification, and generalization, respectively. Through a think-aloud study with 10 participants and a real-world deployment with 1,500+ sessions, the work demonstrates how users transform open-ended tasks into structured classifications, iteratively refine pipelines, and rely on decomposition to manage complex operations. The findings illuminate practical impacts on data analysis workflows and offer design guidance for future human-AI collaborative data systems that integrate data visibility, prompt engineering, and automated decomposition.
Abstract
Unstructured text has long been difficult to automatically analyze at scale. Large language models (LLMs) now offer a way forward by enabling {\em semantic data processing}, where familiar data processing operators (e.g., map, reduce, filter) are powered by LLMs instead of code. However, building effective semantic data processing pipelines presents a departure from traditional data pipelines: users need to understand their data to write effective pipelines, yet they need to construct pipelines to extract the data necessary for that understanding -- all while navigating LLM idiosyncrasies and inconsistencies. We present \docwrangler, a mixed-initiative integrated development environment (IDE) for semantic data processing with three novel features to address the gaps between the user, their data, and their pipeline: {\em (i) In-Situ User Notes} that allows users to inspect, annotate, and track observations across documents and LLM outputs, {\em (ii) LLM-Assisted Prompt Refinement} that transforms user notes into improved operations, and {\em (iii) LLM-Assisted Operation Decomposition} that identifies when operations or documents are too complex for the LLM to correctly process and suggests decompositions. Our evaluation combines a think-aloud study with 10 participants and a public-facing deployment (available at \href{https://docetl.org/playground}{docetl.org/playground}) with 1,500+ recorded sessions, revealing how users develop systematic strategies for their semantic data processing tasks; e.g., transforming open-ended operations into classifiers for easier validation and intentionally using vague prompts to learn more about their data or LLM capabilities.