Vision-based Wearable Steering Assistance for People with Impaired Vision in Jogging

TL;DR

This work introduces Vision-based Wearable Steering Assistance for People with Impaired Vision in Jogging, addressing the gap in fast, independent mobility for BVI athletes on athletics tracks. It combines a lightweight multitask perception network, RunnerNet, with spline-based Frenet-to-Cartesian planning constrained by detected track lines and obstacles, and delivers audio guidance on an embedded Jetson Orin NX. A new MAT dataset of 1,000 athletics-track images enables targeted multi-task learning for track-line segmentation and obstacle detection, with pretraining on BDD100K and fine-tuning on MAT shown to enhance performance. Outdoor tests with blindfolded users demonstrate practical movement on a 400 m track at about 1.34 m/s, indicating the approach’s potential to broaden participation in sports for the visually impaired, while highlighting avenues for faster running speeds in future work.

Abstract

Outdoor sports pose a challenge for people with impaired vision. The demand for higher-speed mobility inspired us to develop a vision-based wearable steering assistance. To ensure broad applicability, we focused on a representative sports environment, the athletics track. Our efforts centered on improving the speed and accuracy of perception, enhancing planning adaptability for the real world, and providing swift and safe assistance for people with impaired vision. In perception, we engineered a lightweight multitask network capable of simultaneously detecting track lines and obstacles. Additionally, due to the limitations of existing datasets for supporting multi-task detection in athletics tracks, we diligently collected and annotated a new dataset (MAT) containing 1000 images. In planning, we integrated the methods of sampling and spline curves, addressing the planning challenges of curves. Meanwhile, we utilized the positions of the track lines and obstacles as constraints to guide people with impaired vision safely along the current track. Our system is deployed on an embedded device, Jetson Orin NX. Through outdoor experiments, it demonstrated adaptability in different sports scenarios, assisting users in achieving free movement of 400-meter at an average speed of 1.34 m/s, meeting the level of normal people in jogging. Our MAT dataset is publicly available from https://github.com/snoopy-l/MAT

Figures (7)

• Figure 1: The hardware of wearable steering assistance. The assistance consists of a pair of RGB-D glasses, an embedded device, and headphones for audio feedback.
• Figure 2: The overview of the wearable steering assistance. The perception module utilizes a multitask network to extract track line and people positions from camera input, which are then used by the path planning module to generate a planned trajectory and provide it as voice feedback.
• Figure 3: Examples of dataset annotations.
• Figure 4: The architecture of RunnerNet. It shares one encoder and combines two decoders to solve different tasks.
• Figure 5: The valuation result trained on different dataset. The yellow ellipses are the false nagative and the red ellipses indicate the false positive.