SmartBook: AI-Assisted Situation Report Generation for Intelligence Analysts

TL;DR

The paper addresses information overload in intelligence analysis by introducing SmartBook, an AI-assisted framework that auto-generates structured situation reports from diverse news sources. It blends a human-centered design process (formative study and collaborative design) with a modular architecture that organizes content into timelines, major events, strategic questions, and grounded claims. Through utility, content review, and editing studies, SmartBook demonstrates high usefulness, trust, and factual grounding, while highlighting that human refinement remains valuable. The work suggests SmartBook can accelerate drafting for analysts and support decision-makers, with planned extensions toward multimodal, multilingual data and reliability verification.

Abstract

Timely and comprehensive understanding of emerging events is crucial for effective decision-making; automating situation report generation can significantly reduce the time, effort, and cost for intelligence analysts. In this work, we identify intelligence analysts' practices and preferences for AI assistance in situation report generation to guide the design strategies for an effective, trust-building interface that aligns with their thought processes and needs. Next, we introduce SmartBook, an automated framework designed to generate situation reports from large volumes of news data, creating structured reports by automatically discovering event-related strategic questions. These reports include multiple hypotheses (claims), summarized and grounded to sources with factual evidence, to promote in-depth situation understanding. Our comprehensive evaluation of SmartBook, encompassing a user study alongside a content review with an editing study, reveals SmartBook's effectiveness in generating accurate and relevant situation reports. Qualitative evaluations indicate over 80% of questions probe for strategic information, and over 90% of summaries produce tactically useful content, being consistently favored over summaries from a large language model integrated with web search. The editing study reveals that minimal information is removed from the generated text (under 2.5%), suggesting that SmartBook provides analysts with a valuable foundation for situation reports

Figures (7)

• Figure 1: Figure showing an example from SmartBoook for the Ukraine-Russia Crisis. SmartBook is organized on a timeline, with 2-week time spans containing chapters and corresponding sections. The section headings are strategic questions and each section content consists of a grounded summary with links to relevant claims, each highlighted for factual evidence and knowledge elements. The italicized entities and events, which comprise the associated knowledge elements, are highlighted in red and blue respectively.
• Figure 2: Storyboard used in the collaborative design sessions with intelligence analysts.
• Figure 3: A screenshot of SmartBook's front-end interface. Within the given situation, the user can navigate timelines (F1), explore strategic questions related to an event (F2), read the overarching summary on a given strategic question (F3), control the depth and length of information (F4), investigate all the claims in the summary (F5), trace each claim to corresponding summary fragment (F6), investigate the source metadata (F7) and read the context from which the claims were extracted (F8).
• Figure 4: Backend workflow for constructing SmartBook. Given the articles corresponding to a specific timeline, the figure shows the process for obtaining the chapters, their section headings, and the corresponding section content.
• Figure 5: Quantitative results from the post-study questionnaire in the utility evaluation.