TapNav: Adaptive Spatiotactile Screen Readers for Tactually Guided Touchscreen Interactions for Blind and Low Vision People

TL;DR

This work tackles the bottleneck of audio-only screen readers for BLV users on touchscreens by introducing TapNav, an adaptive spatiotactile screen reader that couples tactile overlays with context-sensitive speech to anchor spatial understanding. The authors instantiate the concept with two tactile-overlay designs (Braille raised markers and geometric cutouts) and two prototypes (data-visualization exploration and interface navigation) and validate them through formative studies and a 12-participant user study. Findings show that tactile landmarks and spatial constraints enhance localization, speed, and structured exploration, while highlighting marker-design trade-offs and the need for multimodal feedback refinements. The paper contributes practical overlay design guidelines, explorative use cases, and a roadmap for extending adaptive spatiotactile screen readers to broader tasks and potential AI-assisted bindings, aiming to improve accessibility of touchscreen devices for BLV people.

Abstract

Screen readers are audio-based software that Blind and Low Vision (BLV) people use to interact with computing devices, such as tablets and smartphones. Although this technology has significantly improved the accessibility of touchscreen devices, the sequential nature of audio limits the bandwidth of information users can receive and process. We introduce TapNav, an adaptive spatiotactile screen reader prototype developed to interact with touchscreen interfaces spatially. TapNav's screen reader provides adaptive auditory feedback that, in combination with a tactile overlay, conveys spatial information and location of interface elements on-screen. We evaluated TapNav with 12 BLV users who interacted with TapNav to explore a data visualization and interact with a bank transactions application. Our qualitative findings show that touch points and spatially constrained navigation helped users anticipate outcomes for faster exploration, and offload cognitive load to touch. We provide design guidelines for creating tactile overlays for adaptive spatiotactile screen readers and discuss their generalizability beyond our exploratory data analysis and everyday application navigation scenarios.

Figures (5)

• Figure 1: The size of the braille and shape markers tested and used (outlined in blue) for the tactile overlays.
• Figure 2: Participants can start to explore a visualization by requesting a summary (four finger tap) or triggering exploration (long press). Then, to request more information, they can hover their finger over a datapoint, or over a physical marker. To cancel an audio cue, they can tap on the screen with two fingers, and to repeat the previously played audio cue they can tap with three fingers anywhere
• Figure 3: Both exploration mode (on the left) and spatial navigation mode (on the right) in the interface navigation prototype used similar gestures as the Data Visualization prototype but included new swiping gestures to control navigation. The spatial navigation mode also showcases how participants select specific interaction areas by touching a physical marker to constrain their navigation inputs. In this example, when the user touches the Coffee Shop label, the screen reader does not provide any auditory feedback because it is outside the currently selected screen division of the Braille letter K (Column 11). If the user wants to access this label, they would either select Row "D" or Column "F" which have line of sight of this label, or they can go on exploration mode.
• Figure 4: Components of TapNav: BLV users touch a physical overlay placed on a touchscreen device. Then, through gesture recognition, and a spatial mapping engine, the screen reader module generates adaptive auditory feedback. Green modules denote OS-provided components, while orange modules indicate components we implemented for our prototype.
• Figure 5: Participants experienced two prototypes to examine the use of TapNav, our adaptive spatiotactile screen reader: a mock bank transactions application (left) and a scatterplot data visualization (right). A different tactile overlay is placed on a touchscreen device in both scenarios: a Braille strip and a layer with cutouts. Each physical marker in the tactile overlay corresponds to specific digital content to reveal underlying structure to BLV people.