Designing and Evaluating AI Margin Notes in Document Reader Software

TL;DR

The paper investigates integrating AI into document readers via AI margin notes—in-text, anchored annotations powered by LLMs. Through three experiments, it evaluates integration vs chat interfaces, manual vs automatic text selection, and six human–AI involvement techniques. Key findings show that integrated AI margin notes are preferred over separate chat interfaces, manual text selection enhances psychological ownership though increases effort, and higher AI involvement yields faster, easier interactions with mixed effects on comprehension. The results inform design guidelines advocating multiple, configurable AI margin-note styles to balance control, effort, ownership, and learning outcomes.

Abstract

AI capabilities for document reader software are usually presented in separate chat interfaces. We explore integrating AI into document comments, a concept we formalize as AI margin notes. Three design parameters characterize this approach: margin notes are integrated with the text while chat interfaces are not; selecting text for a margin note can be automated through AI or manual; and the generation of a margin note can involve AI to various degrees. Two experiments investigate integration and selection automation, with results showing participants prefer integrated AI margin notes and manual selection. A third experiment explores human and AI involvement through six alternative techniques. Techniques with less AI involvement resulted in more psychological ownership, but faster and less effortful designs are generally preferred. Surprisingly, the degree of AI involvement had no measurable effect on reading comprehension. Our work shows that AI margin notes are desirable and contributes implications for their design.

Figures (10)

• Figure 1: Integration: (a) a chat-based interface is not integrated since it is separated from the associated document text and (b) AI margin notes are integrated since they are associated to specific document text. In this and following two figures: green denotes human-written text, like a prompt, and blue represents AI-generated text, like a response. Yellow indicates text that the AI margin note is linked to.
• Figure 2: Selection automation: (a) text can be automatically selected and multiple AI margin notes created at once and (b) the user can manually select text to create an AI margin note.
• Figure 3: Levels of human and AI involvement: traditional margin notes consist of text that is 100% written by a human, but an AI margin note could consist of text that was 100% generated by AI, or could involve more human-generated text.
• Figure 4: Experimental reading interface: (a) toolbar containing instructions, (b) document to read, and (c) space for specific design variations to be displayed.
• Figure 5: Techniques tested in Experiment 1: (a) a chat-based interface and (b) AI margin notes.