LLM-based In-situ Thought Exchanges for Critical Paper Reading

TL;DR

This work tackles how to bolster critical reading among junior researchers by embedding LLM-based AI agents into an in-situ thought-exchange interface. Through a formative study and a two-week user study with 46 participants, the authors compare no-agent, single-agent, and multi-agent conditions, finding that agent-mediated exchanges significantly boost overall critical-thinking scores and prompt distinct reading practices. Single-agent interactions tended to foster direct text annotations, while multi-agent setups encouraged cross-perspective analysis, with some participants reporting information overload from multiple viewpoints. The study provides design guidance for AI-assisted critical reading tools, showing that AI agents can actively scaffold, rather than offload, higher-order thinking during scholarly reading.

Abstract

Critical reading is a primary way through which researchers develop their critical thinking skills. While exchanging thoughts and opinions with peers can strengthen critical reading, junior researchers often lack access to peers who can offer diverse perspectives. To address this gap, we designed an in-situ thought exchange interface informed by peer feedback from a formative study (N=8) to support junior researchers' critical paper reading. We evaluated the effects of thought exchanges under three conditions (no-agent, single-agent, and multi-agent) with 46 junior researchers over two weeks. Our results showed that incorporating agent-mediated thought exchanges during paper reading significantly improved participants' critical thinking scores compared to the no-agent condition. In the single-agent condition, participants more frequently made reflective annotations on the paper content. In the multi-agent condition, participants engaged more actively with agents' responses. Our qualitative analysis further revealed that participants compared and analyzed multiple perspectives in the multi-agent condition. This work contributes to understanding in-situ AI-based support for critical paper reading through thought exchanges and offers design implications for future research.

Figures (8)

• Figure 1: Formative study has two sessions: (1) left is initial ideation session, participants used Google Doc to read the paper with the experimenter to observe how participants interacted with both the PDF viewer and exchanged thoughts with reading peers; (2) right is iteration session, based on the findings from the first session, we further implemented an initial prototype to gather more feedback.
• Figure 2: The user interface consists of a custom PDF viewer, a left Comment Pane, and a right Section Pane. Users can highlight sentences, leave comments, generate critical thinking questions, and engage in multi-turn thought exchanges with AI agents.
• Figure 3: When users select a sentence (A) that overlaps with a pre-tagged sentence, the system automatically highlights the corresponding pre-tagged sentence (B). Users can then add the sentence to the Comment Pane and maintain the highlight in the PDF viewer (C).
• Figure 4: User study procedure
• Figure 5: External review scores in five dimensions of the Critical Thinking Value Rubric, and their average as total critical thinking scores, across three groups and pre-/post-training tests.