Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

SonoVision: A Computer Vision Approach for Helping Visually Challenged Individuals Locate Objects with the Help of Sound Cues

Md Abu Obaida Zishan, Annajiat Alim Rasel

TL;DR

The paper tackles object localization for visually impaired individuals by embedding CNN-based object detection in a Flutter-based mobile app that conveys spatial cues through stereo audio. SonoVision supports offline operation and a modular framework designed to swap detectors, initially employing EfficientDet-D2 on-device. The key contributions include an end-to-end pipeline from voice-defined objects to real-time auditory guidance and a detailed discussion of architectural modularity and open-set detection limitations. The authors acknowledge currentopen-set coverage gaps and propose future work on lightweight open-set detectors and edge-optimized techniques to enable broader, practical deployment for visually challenged users.

Abstract

Locating objects for the visually impaired is a significant challenge and is something no one can get used to over time. However, this hinders their independence and could push them towards risky and dangerous scenarios. Hence, in the spirit of making the visually challenged more self-sufficient, we present SonoVision, a smart-phone application that helps them find everyday objects using sound cues through earphones/headphones. This simply means, if an object is on the right or left side of a user, the app makes a sinusoidal sound in a user's respective ear through ear/headphones. However, to indicate objects located directly in front, both the left and right earphones are rung simultaneously. These sound cues could easily help a visually impaired individual locate objects with the help of their smartphones and reduce the reliance on people in their surroundings, consequently making them more independent. This application is made with the flutter development platform and uses the Efficientdet-D2 model for object detection in the backend. We believe the app will significantly assist the visually impaired in a safe and user-friendly manner with its capacity to work completely offline. Our application can be accessed here https://github.com/MohammedZ666/SonoVision.git.

SonoVision: A Computer Vision Approach for Helping Visually Challenged Individuals Locate Objects with the Help of Sound Cues

TL;DR

The paper tackles object localization for visually impaired individuals by embedding CNN-based object detection in a Flutter-based mobile app that conveys spatial cues through stereo audio. SonoVision supports offline operation and a modular framework designed to swap detectors, initially employing EfficientDet-D2 on-device. The key contributions include an end-to-end pipeline from voice-defined objects to real-time auditory guidance and a detailed discussion of architectural modularity and open-set detection limitations. The authors acknowledge currentopen-set coverage gaps and propose future work on lightweight open-set detectors and edge-optimized techniques to enable broader, practical deployment for visually challenged users.

Abstract

Locating objects for the visually impaired is a significant challenge and is something no one can get used to over time. However, this hinders their independence and could push them towards risky and dangerous scenarios. Hence, in the spirit of making the visually challenged more self-sufficient, we present SonoVision, a smart-phone application that helps them find everyday objects using sound cues through earphones/headphones. This simply means, if an object is on the right or left side of a user, the app makes a sinusoidal sound in a user's respective ear through ear/headphones. However, to indicate objects located directly in front, both the left and right earphones are rung simultaneously. These sound cues could easily help a visually impaired individual locate objects with the help of their smartphones and reduce the reliance on people in their surroundings, consequently making them more independent. This application is made with the flutter development platform and uses the Efficientdet-D2 model for object detection in the backend. We believe the app will significantly assist the visually impaired in a safe and user-friendly manner with its capacity to work completely offline. Our application can be accessed here https://github.com/MohammedZ666/SonoVision.git.
Paper Structure (15 sections, 5 figures, 1 table)

This paper contains 15 sections, 5 figures, 1 table.

Table of Contents

  1. Introduction
  2. Background and Motivation
  3. Outline
  4. Working methodology
  5. Tech Stack & Application Arhcitecture
  6. Tech Stack
  7. Application Architecture
  8. lib
  9. assets
  10. pubsec.yaml
  11. Architecture, Limitations, and Adaptability of Pretrained Object Detection Models
  12. EfficientDet-D2
  13. Drawbacks:
  14. Adaptability
  15. Conclusion and future work

Figures (5)

  • Figure 1: Our application workflow
  • Figure 2: Figures \ref{['fig:cup_left']}, \ref{['fig:cup_center']}, and \ref{['fig:cup_right']} shows the working principle of our application, indicating the position of an OI (cup) on left, center, and, right sides, through sound cues in the left, both, and right ears respectively (through ear/headphones). The figures also show the active ear/headphone(s) as indicated by the speaker emoji.
  • Figure 3: Directory structure of our application
  • Figure 4: Directory structure of the lib folder
  • Figure 5: Network architecture of EfficientDet model family