Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Text to Blind Motion

Hee Jae Kim, Kathakoli Sengupta, Masaki Kuribayashi, Hernisa Kacorri, Eshed Ohn-Bar

TL;DR

BlindWays addresses the gap in 3D pedestrian motion benchmarks by introducing a multimodal dataset of blind/low-vision pedestrians navigating real urban environments with IMU-based motion capture and rich text descriptions. The dataset comprises 1,029 motion clips, ~0.6M poses, and 2,058 paired textual annotations from 11 participants, collected on eight real-world routes and annotated with high- and low-level language. Experiments show current text-to-motion models struggle to generalize to blind motion, though training on BlindWays improves text–motion alignment and realism, with MDN delivering strong diversity and fidelity. This benchmark enables safer, disability-aware reasoning in autonomous systems and urban planning, and is publicly available for broader research use.

Abstract

People who are blind perceive the world differently than those who are sighted, which can result in distinct motion characteristics. For instance, when crossing at an intersection, blind individuals may have different patterns of movement, such as veering more from a straight path or using touch-based exploration around curbs and obstacles. These behaviors may appear less predictable to motion models embedded in technologies such as autonomous vehicles. Yet, the ability of 3D motion models to capture such behavior has not been previously studied, as existing datasets for 3D human motion currently lack diversity and are biased toward people who are sighted. In this work, we introduce BlindWays, the first multimodal motion benchmark for pedestrians who are blind. We collect 3D motion data using wearable sensors with 11 blind participants navigating eight different routes in a real-world urban setting. Additionally, we provide rich textual descriptions that capture the distinctive movement characteristics of blind pedestrians and their interactions with both the navigation aid (e.g., a white cane or a guide dog) and the environment. We benchmark state-of-the-art 3D human prediction models, finding poor performance with off-the-shelf and pre-training-based methods for our novel task. To contribute toward safer and more reliable systems that can seamlessly reason over diverse human movements in their environments, our text-and-motion benchmark is available at https://blindways.github.io.

Text to Blind Motion

TL;DR

BlindWays addresses the gap in 3D pedestrian motion benchmarks by introducing a multimodal dataset of blind/low-vision pedestrians navigating real urban environments with IMU-based motion capture and rich text descriptions. The dataset comprises 1,029 motion clips, ~0.6M poses, and 2,058 paired textual annotations from 11 participants, collected on eight real-world routes and annotated with high- and low-level language. Experiments show current text-to-motion models struggle to generalize to blind motion, though training on BlindWays improves text–motion alignment and realism, with MDN delivering strong diversity and fidelity. This benchmark enables safer, disability-aware reasoning in autonomous systems and urban planning, and is publicly available for broader research use.

Abstract

People who are blind perceive the world differently than those who are sighted, which can result in distinct motion characteristics. For instance, when crossing at an intersection, blind individuals may have different patterns of movement, such as veering more from a straight path or using touch-based exploration around curbs and obstacles. These behaviors may appear less predictable to motion models embedded in technologies such as autonomous vehicles. Yet, the ability of 3D motion models to capture such behavior has not been previously studied, as existing datasets for 3D human motion currently lack diversity and are biased toward people who are sighted. In this work, we introduce BlindWays, the first multimodal motion benchmark for pedestrians who are blind. We collect 3D motion data using wearable sensors with 11 blind participants navigating eight different routes in a real-world urban setting. Additionally, we provide rich textual descriptions that capture the distinctive movement characteristics of blind pedestrians and their interactions with both the navigation aid (e.g., a white cane or a guide dog) and the environment. We benchmark state-of-the-art 3D human prediction models, finding poor performance with off-the-shelf and pre-training-based methods for our novel task. To contribute toward safer and more reliable systems that can seamlessly reason over diverse human movements in their environments, our text-and-motion benchmark is available at https://blindways.github.io.

Paper Structure

This paper contains 11 sections, 2 figures, 4 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. The BlindWays Dataset
  4. Overview
  5. Data Collection Procedure
  6. Data Annotation Pipeline
  7. Data Analysis
  8. Experiments
  9. Limitations
  10. Conclusion
  11. Acknowledgments

Figures (2)

  • Figure 1: Data Collection with Wearable IMU-based Sensors. Depicting a frame from the study with diverse route stimuli, intersections, a motion capture, and a wide-angle egocentric camera view.
  • Figure 2: Qualitative Examples From Our Dataset. Annotation language captures both high-level information regarding general action, as well as detailed low-level motion characteristics, mobility aid strategies, goals, and environmental context.