Supporting Business Document Workflows via Collection-Centric Information Foraging with Large Language Models

TL;DR

Marco tackles the burden of information foraging over large business document collections by introducing a collection-centric, mixed-initiative workspace that integrates three views—Document View, Notebook View, Table View—and actions (Search, Ask, Summarize) powered by LLMs. The approach reduces cognitive load and speeds sensemaking, validated by a controlled usability study (n=16) showing 16% faster task completion with Marco and reduced effort, without sacrificing accuracy; a design probe with domain experts (n=7) highlights real-world workflow benefits and trust considerations. The system architecture preprocesses PDFs into a collection context, uses per-document extraction driving a Table View, and employs multi-phase prompting (including GPT-4 for collection-level QA) to synthesize across documents. The work contributes design goals for workplace AI-assisted document sensemaking, demonstrates a practical prototype, and discusses implications for reliability, trust, and future multimodal extensions.

Abstract

Knowledge workers often need to extract and analyze information from a collection of documents to solve complex information tasks in the workplace, e.g., hiring managers reviewing resumes or analysts assessing risk in contracts. However, foraging for relevant information can become tedious and repetitive over many documents and criteria of interest. We introduce Marco, a mixed-initiative workspace supporting sensemaking over diverse business document collections. Through collection-centric assistance, Marco reduces the cognitive costs of extracting and structuring information, allowing users to prioritize comparative synthesis and decision making processes. Users interactively communicate their information needs to an AI assistant using natural language and compose schemas that provide an overview of a document collection. Findings from a usability study (n=16) demonstrate that when using Marco, users complete sensemaking tasks 16% more quickly, with less effort, and without diminishing accuracy. A design probe with seven domain experts identifies how Marco can benefit various real-world workflows.

Figures (10)

• Figure 1: Marco is a mixed-initiative workspace for sensemaking over document collections. Marco integrates three views: a Document View renders a PDF document, a Notebook View provides document-centered actions leveraging LLMs, and a Table View provides a collection-level overview. Actions in the Notebook View encode relevant information within result tables, with one row per document (1). Responses can be verified with in-context highlights within the Document View (2). Results across actions are concatenated into a Table View to support collection-level analysis (3).
• Figure 2: (a) The sensemaking process consists of two iterative loops of activity: information foraging and sensemaking pirolli_sensemaking_2005. (b) Marco was designed to support different stages of the sensemaking process in business workflows. Using natural language, users delegate foraging tasks to AI assistance (foraging loop), enabling users to focus on verifying AI responses, refining information schemas, and synthesizing information (sensemaking loop). Solid lines indicate capabilities available in Marco.
• Figure 3: Marco's Notebook View is comprised of various cells. Action cells provide collection-centric AI assistance for users' dynamic information needs. Ask[Each Document] answers the same question for each document separately (1a), Ask[My Collection] answers questions synthesizing information across a collection (1b), Search extracts information verbatim from each document (1c), and Summarize generates a user-guided summary for each document (1d). Text cells serve as a note-taking space (2), and AI Suggestion cells provide follow-up actions to continue the foraging process (3).
• Figure 4: Marco supports four strategies for information foraging across a collection. Search returns snippets extracted directly from each document, Ask[Each Document] and Summarize return LLM-generated answers grounded in each document, and Ask[Each Document] returns a combination of both extracted evidence and an LLM-generated answer synthesizing the evidence.
• Figure 5: Overview of Marco's multi-step pipeline for answering users' queries over a document collection. Marco first identifies attributes required to answer the query, some of which may already have been retrieved by prior user queries and other which may be missing (1). For each missing attribute, Marco executes a search to extract relevant snippets of information from each document (2), and saves the new search results into the aggregate table (3). Search results for each of the required attributes are formatted into a prompt and sent to an LLM (4), whose response to displayed to the user (5).