Empath-D: VR-based Empathetic App Design for Accessibility

TL;DR

Empath-D introduces a VR-based, empathetic design framework that lets designers evaluate mobile apps from the perspective of impaired users by pairing a real smartphone with a VR-driven impaired view. The system uses a 3-tier virtualization with a phone as a tracker, a powerful intermediary for app emulation, and a VR renderer that injects impairment effects, achieving holistic, low-latency, realistic emulation. It demonstrates fidelity to real hardware simulators and real-device interaction, showing comparable accuracy and usability while enabling rapid, customizable impairment profiles (vision and hearing) and multi-modal perturbations. The work provides a practical pathway to more accessible mobile UIs, highlights latency and ergonomics considerations, and outlines future improvements in impairment filters, advanced uses, and latency reductions for broader deployment.

Abstract

With app-based interaction increasingly permeating all aspects of daily living, it is essential to ensure that apps are designed to be \emph{inclusive} and are usable by a wider audience such as the elderly, with various impairments (e.g., visual, audio and motor). We propose \names, a system that fosters empathetic design, by allowing app designers, \emph{in-situ}, to rapidly evaluate the usability of their apps, from the perspective of impaired users. To provide a truly authentic experience, \name carefully orchestrates the interaction between a smartphone and a VR device, allowing the user to experience simulated impairments in a virtual world while interacting naturally with the app, using a real smartphone. By carefully orchestrating the VR-smartphone interaction, \name tackles challenges such as preserving low-latency app interaction, accurate visualization of hand movement and low-overhead perturbation of I/O streams. Experimental results show that user interaction with \name is comparable (both in accuracy and user perception) to real-world app usage, and that it can simulate impairment effects as effectively as a custom hardware simulator.

Figures (12)

• Figure 1: Overview of Empath-D
• Figure 2: Empath-D architecture
• Figure 3: Tracking physical phone with fiducial markers
• Figure 4: Mesh of hand
• Figure 5: Empath-D hand segmentation