Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

MapIO: Embodied Interaction for the Accessibility of Tactile Maps Through Augmented Touch Exploration and Conversation

Matteo Manzoni, Sergio Mascetti, Dragan Ahmetovic, Ryan Crabb, James M. Coughlan

TL;DR

The paper addresses the limited information bandwidth of tactile maps for blind or low-vision users by introducing MapIO, a system that couples embodied tactile exploration with an LLM-driven conversational interface.It presents an iterative, user-centered design process across five phases, including formative studies, preliminary experiments, and a final user study with 10 BLV participants, to tailor interactions and prompts.A core contribution is the prompt augmentation engineering workflow and the use of local tool calls (e.g., routing APIs, spatial reasoning tools) to compensate for LLM spatial reasoning gaps, achieving high accuracy in map-question answering.The work demonstrates that BLV users can perform complex tasks—exploration, navigation, and POI queries—through natural pointing and conversation, with SUS scores indicating good usability and room for performance improvements as LLMs evolve.

Abstract

For individuals who are blind or have low vision, tactile maps provide essential spatial information but are limited in the amount of data they can convey. Digitally augmented tactile maps enhance these capabilities with audio feedback, thereby combining the tactile feedback provided by the map with an audio description of the touched elements. In this context, we explore an embodied interaction paradigm to augment tactile maps with conversational interaction based on Large Language Models, thus enabling users to obtain answers to arbitrary questions regarding the map. We analyze the type of questions the users are interested in asking, engineer the Large Language Model's prompt to provide reliable answers, and study the resulting system with a set of 10 participants, evaluating how the users interact with the system, its usability, and user experience.

MapIO: Embodied Interaction for the Accessibility of Tactile Maps Through Augmented Touch Exploration and Conversation

TL;DR

The paper addresses the limited information bandwidth of tactile maps for blind or low-vision users by introducing MapIO, a system that couples embodied tactile exploration with an LLM-driven conversational interface.It presents an iterative, user-centered design process across five phases, including formative studies, preliminary experiments, and a final user study with 10 BLV participants, to tailor interactions and prompts.A core contribution is the prompt augmentation engineering workflow and the use of local tool calls (e.g., routing APIs, spatial reasoning tools) to compensate for LLM spatial reasoning gaps, achieving high accuracy in map-question answering.The work demonstrates that BLV users can perform complex tasks—exploration, navigation, and POI queries—through natural pointing and conversation, with SUS scores indicating good usability and room for performance improvements as LLMs evolve.

Abstract

For individuals who are blind or have low vision, tactile maps provide essential spatial information but are limited in the amount of data they can convey. Digitally augmented tactile maps enhance these capabilities with audio feedback, thereby combining the tactile feedback provided by the map with an audio description of the touched elements. In this context, we explore an embodied interaction paradigm to augment tactile maps with conversational interaction based on Large Language Models, thus enabling users to obtain answers to arbitrary questions regarding the map. We analyze the type of questions the users are interested in asking, engineer the Large Language Model's prompt to provide reliable answers, and study the resulting system with a set of 10 participants, evaluating how the users interact with the system, its usability, and user experience.

Paper Structure

This paper contains 62 sections, 5 figures, 5 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Digitally Augmented Tactile Supports
  4. Use of conversational interfaces in Assistive Technologies
  5. Interaction and System Design
  6. Formative questionnaire
  7. Initial system and interaction design
  8. System Design
  9. Interaction Design
  10. First preliminary study
  11. Study design
  12. Results
  13. Outcomes
  14. Second Preliminary Study
  15. Study Design
  16. ...and 47 more sections

Figures (5)

  • Figure 1: MapIO initial design architecture
  • Figure 2: Data flow involving the LLM with focus on prompt augmentation.
  • Figure 3: LLM query context engineering pipeline
  • Figure 4: Participant shown interacting with MapIO, with Bluetooth buttons nearby.
  • Figure 5: SUS item results, aggregated over participants ($\bullet$ Mean, --- Median, - - Benchmark Average, $\cdot \cdot \cdot$ Benchmark Good)