Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Proactive Agentic Whiteboards: Enhancing Diagrammatic Learning

Suveen Ellawela, Sashenka Gamage, Dinithi Dissanayake

TL;DR

Proactive Agentic Whiteboards address the cognitive load of real-time diagram creation by listening to spoken explanations and proactively refining diagrams. The authors implement DrawDash as a TAB-like interface on a tldraw-based whiteboard, using Gemini-2.5 for image updates and speech-to-text for intent detection. They demonstrate four domain cases (computer science, web development, biology, chemistry) to illustrate improved diagram clarity and potential pedagogy benefits, while noting limitations and the need for formal classroom evaluation. The work contributes a concrete architecture and design principles for speech-driven, proactive diagram augmentation in education.

Abstract

Educators frequently rely on diagrams to explain complex concepts during lectures, yet creating clear and complete visual representations in real time while simultaneously speaking can be cognitively demanding. Incomplete or unclear diagrams may hinder student comprehension, as learners must mentally reconstruct missing information while following the verbal explanation. Inspired by advances in code completion tools, we introduce DrawDash, an AI-powered whiteboard assistant that proactively completes and refines educational diagrams through multimodal understanding. DrawDash adopts a TAB-completion interaction model: it listens to spoken explanations, detects intent, and dynamically suggests refinements that can be accepted with a single keystroke. We demonstrate DrawDash across four diverse teaching scenarios, spanning topics from computer science and web development to biology. This work represents an early exploration into reducing instructors' cognitive load and improving diagram-based pedagogy through real-time, speech-driven visual assistance, and concludes with a discussion of current limitations and directions for formal classroom evaluation.

Proactive Agentic Whiteboards: Enhancing Diagrammatic Learning

TL;DR

Proactive Agentic Whiteboards address the cognitive load of real-time diagram creation by listening to spoken explanations and proactively refining diagrams. The authors implement DrawDash as a TAB-like interface on a tldraw-based whiteboard, using Gemini-2.5 for image updates and speech-to-text for intent detection. They demonstrate four domain cases (computer science, web development, biology, chemistry) to illustrate improved diagram clarity and potential pedagogy benefits, while noting limitations and the need for formal classroom evaluation. The work contributes a concrete architecture and design principles for speech-driven, proactive diagram augmentation in education.

Abstract

Educators frequently rely on diagrams to explain complex concepts during lectures, yet creating clear and complete visual representations in real time while simultaneously speaking can be cognitively demanding. Incomplete or unclear diagrams may hinder student comprehension, as learners must mentally reconstruct missing information while following the verbal explanation. Inspired by advances in code completion tools, we introduce DrawDash, an AI-powered whiteboard assistant that proactively completes and refines educational diagrams through multimodal understanding. DrawDash adopts a TAB-completion interaction model: it listens to spoken explanations, detects intent, and dynamically suggests refinements that can be accepted with a single keystroke. We demonstrate DrawDash across four diverse teaching scenarios, spanning topics from computer science and web development to biology. This work represents an early exploration into reducing instructors' cognitive load and improving diagram-based pedagogy through real-time, speech-driven visual assistance, and concludes with a discussion of current limitations and directions for formal classroom evaluation.

Paper Structure

This paper contains 15 sections, 5 figures.

Table of Contents

  1. Introduction
  2. Related Work
  3. Proactive, agentic assistants in interactive workflows
  4. AI-enhanced whiteboards and multimodal canvases
  5. Diagrammatic instruction, progressive drawing, and multimodal learning evidence
  6. Intelligent Autocompletion and Agentic Collaboration in Learning
  7. Methodology
  8. System Workflow and Integration
  9. System Demonstrations
  10. Methodology
  11. Case 1: Binary Search Tree (Computer Science)
  12. Case 2: HTTP Request Flow (Web Development)
  13. Case 3: Photosynthesis Process (Biology)
  14. Case 4: Heating Liquid in a Beaker (Chemistry)
  15. Limitations and Future Work

Figures (5)

  • Figure 1: System architecture of DrawDash showing the integration of speech processing, visual analysis, and real-time diagram suggestion modules.
  • Figure 2: Binary Search Tree completion example. (Top:) Instructor's initial sketch showing only the root node and partial child links. (Bottom:) DrawDash's suggested completion illustrating the full binary search tree structure with balanced branches.
  • Figure 3: HTTP request flow example. (Top:) Instructor's initial sketch showing partial browser-server connection. (Bottom:) DrawDash's suggested completion illustrating the full browser–server–database interaction cycle.
  • Figure 4: Photosynthesis process example. (Top:) Instructor's initial sketch showing a leaf with partial input labels. (Bottom:) DrawDash's suggested completion illustrating the full process of photosynthesis, including inputs (CO2, H2O, sunlight) and outputs (glucose, O2).
  • Figure 5: Heating liquid in a beaker example. (Top:) Instructor's initial sketch showing a beaker with liquid and a thermometer. (Bottom:) DrawDash's suggested completion illustrating the heating process with a Bunsen burner and rising bubbles indicating the onset of boiling.