HACI: A Haptic-Audio Code Interface to Improve Educational Outcomes for Visually Impaired Introductory Programming Students

TL;DR

The paper presents HACI, a haptic-audio interface embedded in a web-based JavaScript IDE to make introductory programming accessible to visually impaired students. It integrates a six-motor glove controlled via the Web Serial API with audio narration and non-speech cues, enabling non-visual code navigation, comprehension, editing, and debugging. A pilot study with ten undergraduates provides preliminary evidence of reduced cognitive load and improved navigation and understanding, while highlighting challenges in feedback consistency, timing, and feature richness. The work contributes a practical blueprint for multimodal accessibility in programming education and motivates future developments—language expansion, AI-assisted feedback, and curricular integration—to broaden inclusion in CS education.

Abstract

This thesis introduces the Haptic-Audio Code Interface (HACI), an educational tool designed to enhance programming education for visually impaired (VI) students by integrating haptic and audio feedback to compensate for the absence of visual cues. HACI consists of a non-resource-intensive web application supporting JavaScript program development, execution, and debugging, connected via a cable to an Arduino-powered glove with six integrated haptic motors to provide physical feedback to VI programmers. Motivated by the need to provide equitable educational opportunities in computer science, HACI aims to improve non-visual code navigation, comprehension, summarizing, editing, and debugging for students with visual impairments while minimizing cognitive load. This work details HACI's design principles, technical implementation, and a preliminary evaluation through a pilot study conducted with undergraduate Computer Science students. Findings indicate that HACI aids in the non-visual navigation and understanding of programming constructs, although challenges remain in refining feedback mechanisms to ensure consistency and reliability, as well as supplementing the current functionality with a more feature-reach and customizable accessible learning experience which will allow visually impaired students to fully utilize interleaved haptic and audio feedback. The study underscores the transformative potential of haptic and audio feedback in educational practices for the visually impaired, setting a foundation for future research and development in accessible programming education. This thesis contributes to the field of accessible technology by demonstrating how tactile and auditory feedback can be effectively integrated into educational tools, thereby broadening accessibility in STEM education.

Figures (7)

• Figure 1: Complete HACI hardware setup. Note the shorter wires connecting the glove motors to the hardware, which are for testing purposes and were elongated to allow comfortable and free hand movement during experimental use.
• Figure 2: Screenshot of HACI user interface with example code, a sample error, and sample console output.
• Figure 3: Fritzing schematic of HACI hardware, including Arduino Mega, I$^2$C multiplexer, 6x DRV2605L haptic motor controllers, and 6 haptic motors.
• Figure 4: Average Task Completion Time by Shortcut Usage.
• Figure 5: Average Number of Errors Raised by Shortcut Usage.