Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Exploring the Design Space of Real-time LLM Knowledge Support Systems: A Case Study of Jargon Explanations

Yuhan Liu, Aadit Shah, Jordan Ackerman, Manaswi Saha

TL;DR

This work investigates how to design real-time LLM-based knowledge support systems to close jargon-induced knowledge gaps in cross-domain communication. It introduces StopGap, a prototype that delivers real-time jargon explanations in multiple formats while playing a video, and conducts a design probe with 24 participants to compare representation formats and interaction modes. The authors articulate a six-dimension design space (target user, representation format, data source, display mode, customization, interaction mode) and discuss credibility versus automation, user agency, and personalization as core design considerations. Findings indicate StopGap is useful and does not impose excessive cognitive load, but user preferences for representations and interactions are highly individual, suggesting future work should support mixed-initiative, customizable designs and explicit source verification to broaden applicability beyond jargon to other knowledge-gap types.

Abstract

Knowledge gaps often arise during communication due to diverse backgrounds, knowledge bases, and vocabularies. With recent LLM developments, providing real-time knowledge support is increasingly viable, but is challenging due to shared and individual cognitive limitations (e.g., attention, memory, and comprehension) and the difficulty in understanding the user's context and internal knowledge. To address these challenges, we explore the key question of understanding how people want to receive real-time knowledge support. We built StopGap -- a prototype that provides real-time knowledge support for explaining jargon words in videos -- to conduct a design probe study (N=24) that explored multiple visual knowledge representation formats. Our study revealed individual differences in preferred representations and highlighted the importance of user agency, personalization, and mixed-initiative assistance. Based on our findings, we map out six key design dimensions for real-time LLM knowledge support systems and offer insights for future research in this space.

Exploring the Design Space of Real-time LLM Knowledge Support Systems: A Case Study of Jargon Explanations

TL;DR

This work investigates how to design real-time LLM-based knowledge support systems to close jargon-induced knowledge gaps in cross-domain communication. It introduces StopGap, a prototype that delivers real-time jargon explanations in multiple formats while playing a video, and conducts a design probe with 24 participants to compare representation formats and interaction modes. The authors articulate a six-dimension design space (target user, representation format, data source, display mode, customization, interaction mode) and discuss credibility versus automation, user agency, and personalization as core design considerations. Findings indicate StopGap is useful and does not impose excessive cognitive load, but user preferences for representations and interactions are highly individual, suggesting future work should support mixed-initiative, customizable designs and explicit source verification to broaden applicability beyond jargon to other knowledge-gap types.

Abstract

Knowledge gaps often arise during communication due to diverse backgrounds, knowledge bases, and vocabularies. With recent LLM developments, providing real-time knowledge support is increasingly viable, but is challenging due to shared and individual cognitive limitations (e.g., attention, memory, and comprehension) and the difficulty in understanding the user's context and internal knowledge. To address these challenges, we explore the key question of understanding how people want to receive real-time knowledge support. We built StopGap -- a prototype that provides real-time knowledge support for explaining jargon words in videos -- to conduct a design probe study (N=24) that explored multiple visual knowledge representation formats. Our study revealed individual differences in preferred representations and highlighted the importance of user agency, personalization, and mixed-initiative assistance. Based on our findings, we map out six key design dimensions for real-time LLM knowledge support systems and offer insights for future research in this space.

Paper Structure

This paper contains 53 sections, 15 figures, 4 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Knowledge & Knowledge Gaps
  4. Knowledge Management and Support Systems
  5. Components in Knowledge Support Systems
  6. Knowledge Representation Formats
  7. Real-time Support in Computer-Mediated Communication
  8. StopGap System
  9. System Description
  10. Design Considerations
  11. System Implementation
  12. Design Probe Study
  13. Study Methodology
  14. Study Materials
  15. Study Video Selection
  16. ...and 38 more sections

Figures (15)

  • Figure 1: StopGap Prototype. To use this system, a user uploads a video. The system auto-transcribes and batch-identifies jargon words. The system then displays the real-time knowledge support in a side panel as flashcards once the user starts playing the video. The flashcards scroll up automatically for every new jargon word.
  • Figure 2: Chosen Knowledge Representation Formats.
  • Figure 3: Mixed Study Design
  • Figure 4: Comparison of participants' task load index of control task and StopGap task
  • Figure 5: Design Space of Real-time Knowledge Support Systems
  • ...and 10 more figures