Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Seeing to Think? How Source Transparency Design Shapes Interactive Information Seeking and Evaluation in Conversational AI

Jiangen He, Jiqun Liu

TL;DR

The paper investigates how source transparency design in conversational AI affects interactive information seeking and evaluation. Through a controlled between-subjects experiment (N=372) comparing four interfaces, it shows that visibility and accessibility of sources reshape exploration strategies and evidence integration, with Hover Card enabling on-demand verification and Aligned Sidebar mitigating information overload as citation density rises. A key finding is the trade-off between workflow fluency and reflective verification, and the evidence suggests adaptive interfaces that balance reading, verification, and synthesis can foster more responsible and critical AI-assisted writing. The work advances practical guidance for designing citation-aware, cognitively supportive conversational AI that scales critical thinking with information density.

Abstract

Conversational AI systems increasingly function as primary interfaces for information seeking, yet how they present sources to support information evaluation remains under-explored. This paper investigates how source transparency design shapes interactive information seeking, trust, and critical engagement. We conducted a controlled between-subjects experiment (N=372) comparing four source presentation interfaces - Collapsible, Hover Card, Footer, and Aligned Sidebar - varying in visibility and accessibility. Using fine-grained behavioral analysis and automated critical thinking assessment, we found that interface design fundamentally alters exploration strategies and evidence integration. While the Hover Card interface facilitated seamless, on-demand verification during the task, the Aligned Sidebar uniquely mitigated the negative effects of information overload: as citation density increased, Sidebar users demonstrated significantly higher critical thinking and synthesis scores compared to other conditions. Our results highlight a trade-off between designs that support workflow fluency and those that enforce reflective verification, offering practical implications for designing adaptive and responsible conversational AI that fosters critical engagement with AI generated content.

Seeing to Think? How Source Transparency Design Shapes Interactive Information Seeking and Evaluation in Conversational AI

TL;DR

The paper investigates how source transparency design in conversational AI affects interactive information seeking and evaluation. Through a controlled between-subjects experiment (N=372) comparing four interfaces, it shows that visibility and accessibility of sources reshape exploration strategies and evidence integration, with Hover Card enabling on-demand verification and Aligned Sidebar mitigating information overload as citation density rises. A key finding is the trade-off between workflow fluency and reflective verification, and the evidence suggests adaptive interfaces that balance reading, verification, and synthesis can foster more responsible and critical AI-assisted writing. The work advances practical guidance for designing citation-aware, cognitively supportive conversational AI that scales critical thinking with information density.

Abstract

Conversational AI systems increasingly function as primary interfaces for information seeking, yet how they present sources to support information evaluation remains under-explored. This paper investigates how source transparency design shapes interactive information seeking, trust, and critical engagement. We conducted a controlled between-subjects experiment (N=372) comparing four source presentation interfaces - Collapsible, Hover Card, Footer, and Aligned Sidebar - varying in visibility and accessibility. Using fine-grained behavioral analysis and automated critical thinking assessment, we found that interface design fundamentally alters exploration strategies and evidence integration. While the Hover Card interface facilitated seamless, on-demand verification during the task, the Aligned Sidebar uniquely mitigated the negative effects of information overload: as citation density increased, Sidebar users demonstrated significantly higher critical thinking and synthesis scores compared to other conditions. Our results highlight a trade-off between designs that support workflow fluency and those that enforce reflective verification, offering practical implications for designing adaptive and responsible conversational AI that fosters critical engagement with AI generated content.
Paper Structure (47 sections, 4 figures, 8 tables)

This paper contains 47 sections, 4 figures, 8 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Conversational Information Seeking and Evaluation
  4. Transparency and Explanation in Human-AI Interaction
  5. Interface Design in User Interaction with Generated Contents
  6. Methods
  7. Interface Design
  8. Collapsible Interface
  9. Hover Card Interface
  10. Footer Interface
  11. Aligned Sidebar Interface
  12. Citing Sources
  13. AI Agent
  14. Procedure
  15. Data Collection
  16. ...and 32 more sections

Figures (4)

  • Figure 1: Interface conditions
  • Figure 2: Behavioral metrics across message rounds (1-6). The charts display trends across the four interface conditions.
  • Figure 3: Visualizing high-probability transitions. Hover condition (top-right) shows strong links between Exploration, Citation, and Writing.
  • Figure 4: Interaction effect of UI design and information density (number of citations) on critical thinking scores. Shaded areas represent 90% confidence intervals.