Traceable Text: Deepening Reading of AI-Generated Summaries with Phrase-Level Provenance Links

TL;DR

A simple interaction primitive, traceable text is designed to support critical examination of generated summaries and the source texts they were derived from to help understand the veracity of those summaries.

Abstract

As AI-generated summaries proliferate, how can we help people understand the veracity of those summaries? In this short paper, we design a simple interaction primitive, traceable text, to support critical examination of generated summaries and the source texts they were derived from. In a traceable text, passages of a generated summary link to passages of the source text that informed them. A traceable text can be generated with a straightforward prompt chaining approach, and optionally adjusted by human authors depending on application. In a usability study, we examined the impact of traceable texts on reading and understanding patient medical records. Traceable text helped readers answer questions about the content of the source text more quickly and markedly improved correctness of answers in cases where there were hallucinations in the summaries. When asked to read a text of personal importance with traceable text, readers employed traceable text as an understanding aid and as an index into the source note.

Figures (6)

• Figure 1: Inspecting hallucinations. When a generated summary contains a hallucination (e.g., the contradiction between dark blue passages in the summary and source above), a traceable text sometimes supports resolution of the contradiction by linking between the most closely related content in the hallucinatory phrase and the original note. This can be particularly useful when contradictions are subtle to the particular reader (as in the example above if read by a non-expert patient) and might otherwise go undetected.
• Figure 2: Backlinking from source to summary. Readers can receive lightweight help in understanding the source text by hovering over a passage in the source text, and seeing the corresponding passage in the summary highlighted.
• Figure 3: Showing all backlinks. The passages of the source that are summarized are not highlighted, to avoid overwhelming the reader. If the reader wants to see passages in the source that link to the summary, they can hold a modal key, and all linked source passages become highlighted.
• Figure 4: Prompt chain for generating traceable text. The chain generates a summary, splits it into claims, and aligns those claims with source passages.
• Figure 5: Correctness. Readers answered questions correctly more often with traceable text than with the baseline. This effect was pronounced and significant for questions about summaries with hallucinations, and not statistically significant for questions about expert-validated summaries.