RASSAR: Room Accessibility and Safety Scanning in Augmented Reality

TL;DR

RASSAR presents a mobile AR system that semiautomatically audits indoor safety and accessibility by fusing LiDAR-based reconstruction with real-time computer vision. It introduces an extensible JSON rubric framework and supports audio accessibility, enabling user-in-the-loop identification and visualization of issues across four categories. Across a formative study, technical evaluation in ten homes, and a user study with six stakeholders, RASSAR achieves competitive detection performance (average precision ~0.86, recall ~0.83) and substantial efficiency gains (≈3.5x faster than manual audits). The work highlights practical deployment scenarios, demonstrates extensibility beyond ADA standards, and provides datasets and open-source resources to accelerate indoor accessibility tooling. Overall, RASSAR advances wearable/mobile AR-assisted home auditing by combining automated detection with user customization and accessible visualization.

Abstract

The safety and accessibility of our homes is critical to quality of life and evolves as we age, become ill, host guests, or experience life events such as having children. Researchers and health professionals have created assessment instruments such as checklists that enable homeowners and trained experts to identify and mitigate safety and access issues. With advances in computer vision, augmented reality (AR), and mobile sensors, new approaches are now possible. We introduce RASSAR, a mobile AR application for semi-automatically identifying, localizing, and visualizing indoor accessibility and safety issues such as an inaccessible table height or unsafe loose rugs using LiDAR and real-time computer vision. We present findings from three studies: a formative study with 18 participants across five stakeholder groups to inform the design of RASSAR, a technical performance evaluation across ten homes demonstrating state-of-the-art performance, and a user study with six stakeholders. We close with a discussion of future AI-based indoor accessibility assessment tools, RASSAR's extensibility, and key application scenarios.

Figures (7)

• Figure 1: Design probes for RASSAR's interaction and interface design. For each question, we provide three or four options as interface mock-ups.
• Figure 2: RASSAR system overview. RASSAR (1) scans and reconstructs a home space, (2) detects accessibility and safety issues in the space, and (3) visualizes real-time results in AR.
• Figure 3: Informed by literature and our formative study, RASSAR can detect 20 types of accessibility and safety issues across four categories: object dimension, object position, risky item, and assistive item. Each issue has relevance to specific accessibility communities, marked with black icons. We acknowledge that safety/accessibility issues can be fluid marked not just by (dis)ability but fatigue, time-of-day, etc. and that individuals may not map exactly to these categories. Our custom JSON-based rubric could allow for precise individual specification in the future (e.g., with a custom authoring interface.
• Figure 4: RASSAR's AR-based scanning interface and a detailed view of a detected issue. (1) During a scan, RASSAR shows detected problems in real-time via AR overlays, including (a) red spheres, which can be selected to view more information and to confirm/delete detection and (b) CV-based detections with green bounding boxes, a text label, and confidence score. To aid understanding of the CV field-of-view, we draw a (c) gray bounding box. We also show a mini-map (d) that adapts to users' orientation/position with real-time reconstruction results. (2) The user can click on identified issues (a) to view more information, see recommended solutions, and to (e) confirm/delete problems.
• Figure 5: Three examples of the post-scan interactive summary of results. A user can interact with the 3D reconstruction, inspect detailed information about objects or issues, and remove any errant or disagreed upon issues. The top-right button lets user export scan results to a JSON file.