Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

MateRobot: Material Recognition in Wearable Robotics for People with Visual Impairments

Junwei Zheng, Jiaming Zhang, Kailun Yang, Kunyu Peng, Rainer Stiefelhagen

TL;DR

A wearable vision-based robotic system, MATERobot, is established for PVI to recognize materials and object categories beforehand, and a lightweight yet accurate model MATEViT is proposed to perform pixel-wise semantic segmentation, simultaneously recognizing both objects and materials.

Abstract

People with Visual Impairments (PVI) typically recognize objects through haptic perception. Knowing objects and materials before touching is desired by the target users but under-explored in the field of human-centered robotics. To fill this gap, in this work, a wearable vision-based robotic system, MateRobot, is established for PVI to recognize materials and object categories beforehand. To address the computational constraints of mobile platforms, we propose a lightweight yet accurate model MateViT to perform pixel-wise semantic segmentation, simultaneously recognizing both objects and materials. Our methods achieve respective 40.2% and 51.1% of mIoU on COCOStuff-10K and DMS datasets, surpassing the previous method with +5.7% and +7.0% gains. Moreover, on the field test with participants, our wearable system reaches a score of 28 in the NASA-Task Load Index, indicating low cognitive demands and ease of use. Our MateRobot demonstrates the feasibility of recognizing material property through visual cues and offers a promising step towards improving the functionality of wearable robots for PVI. The source code has been made publicly available at https://junweizheng93.github.io/publications/MATERobot/MATERobot.html.

MateRobot: Material Recognition in Wearable Robotics for People with Visual Impairments

TL;DR

A wearable vision-based robotic system, MATERobot, is established for PVI to recognize materials and object categories beforehand, and a lightweight yet accurate model MATEViT is proposed to perform pixel-wise semantic segmentation, simultaneously recognizing both objects and materials.

Abstract

People with Visual Impairments (PVI) typically recognize objects through haptic perception. Knowing objects and materials before touching is desired by the target users but under-explored in the field of human-centered robotics. To fill this gap, in this work, a wearable vision-based robotic system, MateRobot, is established for PVI to recognize materials and object categories beforehand. To address the computational constraints of mobile platforms, we propose a lightweight yet accurate model MateViT to perform pixel-wise semantic segmentation, simultaneously recognizing both objects and materials. Our methods achieve respective 40.2% and 51.1% of mIoU on COCOStuff-10K and DMS datasets, surpassing the previous method with +5.7% and +7.0% gains. Moreover, on the field test with participants, our wearable system reaches a score of 28 in the NASA-Task Load Index, indicating low cognitive demands and ease of use. Our MateRobot demonstrates the feasibility of recognizing material property through visual cues and offers a promising step towards improving the functionality of wearable robots for PVI. The source code has been made publicly available at https://junweizheng93.github.io/publications/MATERobot/MATERobot.html.
Paper Structure (20 sections, 2 equations, 14 figures, 3 tables)

This paper contains 20 sections, 2 equations, 14 figures, 3 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Wearable Assistive System
  4. Material Semantic Segmentation
  5. MateRobot: A Wearable Robotic System
  6. Hardware Component
  7. User Interaction
  8. MateViT Model
  9. Learnable Importance Sampling
  10. Multi-gate Mixture-of-Experts
  11. Experiments
  12. Settings and Datasets
  13. Results on Material Segmentation
  14. Results on Multi-task Learning
  15. Ablation Study
  16. ...and 5 more sections

Figures (14)

  • Figure 1: Wearable robotic system
  • Figure 2: Predict material & object
  • Figure 4: Glasses and processor
  • Figure 5: Components in glasses
  • Figure 7: Architecture of MateViT in a Learnable Importance Sampling (LIS) strategy to reduce the computational complexity, and with a Multi-gate Mixture-of-Experts (MMoE) layer to perform dual-task segmentation (i.e., #1-Object and #2-Material segmentation).
  • ...and 9 more figures