Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

'Show It, Don't Just Say It': The Complementary Effects of Instruction Multimodality for Software Guidance

Emran Poh, Yueyue Hou, Tianyi Zhang, Jiannan Li

TL;DR

An observational study of human teacher-student pairs, where experienced design software users taught novices two new graphic design software features through multi-step procedures, finds that annotations complement speech with spatial precision and remote control complements it with spatial and temporal precision, but both cause intrusion to learner agency.

Abstract

Designing adaptive tutoring systems for software learning presents challenges in determining appropriate instructional modalities. To inform the design of such systems, we conducted an observational study of ten human teacher-student pairs (N=10), where experienced design software users taught novices two new graphic design software features through multi-step procedures. These lessons were limited to three communication channels (speech, visual annotations, and remote screen control) to mimic possible AI tutor modalities. We found that annotations complement speech with spatial precision and remote control complements it with spatial and temporal precision, but both cause intrusion to learner agency. Teachers adaptively select modalities to balance the need for instruction progress with students' cognitive engagement and sense of digital territory ownership. Our results provide further support to the contiguity principles and the value of agency in learning, while suggesting precision-agency trade-off and digital territoriality as new design constraints for adaptive software guidance.

'Show It, Don't Just Say It': The Complementary Effects of Instruction Multimodality for Software Guidance

TL;DR

An observational study of human teacher-student pairs, where experienced design software users taught novices two new graphic design software features through multi-step procedures, finds that annotations complement speech with spatial precision and remote control complements it with spatial and temporal precision, but both cause intrusion to learner agency.

Abstract

Designing adaptive tutoring systems for software learning presents challenges in determining appropriate instructional modalities. To inform the design of such systems, we conducted an observational study of ten human teacher-student pairs (N=10), where experienced design software users taught novices two new graphic design software features through multi-step procedures. These lessons were limited to three communication channels (speech, visual annotations, and remote screen control) to mimic possible AI tutor modalities. We found that annotations complement speech with spatial precision and remote control complements it with spatial and temporal precision, but both cause intrusion to learner agency. Teachers adaptively select modalities to balance the need for instruction progress with students' cognitive engagement and sense of digital territory ownership. Our results provide further support to the contiguity principles and the value of agency in learning, while suggesting precision-agency trade-off and digital territoriality as new design constraints for adaptive software guidance.
Paper Structure (56 sections, 7 figures, 4 tables)

This paper contains 56 sections, 7 figures, 4 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Foundational Learning Theories
  4. Software Task Assistance and Learning
  5. Multimodality in Education
  6. Adaptiveness of Guidance Tools
  7. Methodology
  8. Participants
  9. Experiment Design and Environment
  10. Procedure
  11. Task Design
  12. Data Collection and Analysis
  13. Results
  14. Study Overview and Participant Performance
  15. Learning Outcomes and Skill Transfer
  16. ...and 41 more sections

Figures (7)

  • Figure 1: Two Figma design tasks: Task 1: Weather Icons Creation (left) and Task 2: Weather Cards Design (right), showing progression from fundamental skills to advanced component usage
  • Figure 2: Annotation Styles and Their Characteristics
  • Figure 3: Teacher demonstrating radial pattern creation through hands-on interface manipulation: Shows step-by-step duplication (Ctrl+C/Ctrl+V) and constrained rotation (Shift+15°) of rectangular objects around a central circle, with accompanying verbal guidance explaining each action.
  • Figure 4: Design implications for AI tutoring systems: (a) Ghost Cursor - showing AI tutor's cursor position to students, (b) Fading Annotations - annotations that gradually disappear over time, (c) Timeline Scrubbing - reviewing past annotations through temporal navigation, and (d) Interactive References - extractable and manipulable content elements from reference images.
  • Figure 5: Overview of the two study tasks used in the teaching sessions: Task 1 (Weather Icons) and Task 2 (Weather Cards).
  • ...and 2 more figures