Anomaly Detection for People with Visual Impairments Using an Egocentric 360-Degree Camera

TL;DR

This work proposes the first step towards detecting anomalous situations for visually impaired people by observing their entire surroundings using an egocentric 360-degree camera, and proposes a new architecture called the FDPN (Frame and Direction Prediction Network), which facilitates frame-level prediction of abnormal events and identifying of their directions.

Abstract

Recent advancements in computer vision have led to a renewed interest in developing assistive technologies for individuals with visual impairments. Although extensive research has been conducted in the field of computer vision-based assistive technologies, most of the focus has been on understanding contexts in images, rather than addressing their physical safety and security concerns. To address this challenge, we propose the first step towards detecting anomalous situations for visually impaired people by observing their entire surroundings using an egocentric 360-degree camera. We first introduce a novel egocentric 360-degree video dataset called VIEW360 (Visually Impaired Equipped with Wearable 360-degree camera), which contains abnormal activities that visually impaired individuals may encounter, such as shoulder surfing and pickpocketing. Furthermore, we propose a new architecture called the FDPN (Frame and Direction Prediction Network), which facilitates frame-level prediction of abnormal events and identifying of their directions. Finally, we evaluate our approach on our VIEW360 dataset and the publicly available UCF-Crime and Shanghaitech datasets, demonstrating state-of-the-art performance.

Figures (9)

• Figure 1: This paper aims to tackle safety and security concerns faced by visually impaired individuals. To tackle these concerns, we introduce a new dataset, VIEW360, specifically designed for detecting unusual activities by observing their entire surroundings using an egocentric 360-degree camera. The dataset is collected through a process involving (a) capturing footage with a wearable 360-degree camera worn around the neck, (b) recording egocentric 360-degree videos to encompass the wearer's surroundings, and (c) stitching these videos into panoramic views for comprehensive analysis. In the depicted scene, the individual highlighted in magenta is attempting a wallet theft.
• Figure 2: This figure contrasts anomaly scores at event start and end boundaries for state-of-the-art method MGFN and our FDPN on VIEW360 dataset. MGFN often makes false predictions at event boundaries because it predicts at the snippet-level, whereas our proposed method makes better predictions at the event boundaries since it can make frame-level predictions.
• Figure 3: Here are some abnormal instances in our VIEW360 dataset. The first row shows theft of personal belongings from the camera-wearer. The second row depicts shoulder-surfing attacks: someone covertly observing the camera wearer's ATM use and smartphone without their awareness. The third row portrays a person with visual impairments being mocked or harassed.
• Figure 4: Distribution of the VIEW360 dataset, illustrating training/testing splits, video locations, and abnormal event orientations. Includes a bar chart of normal and abnormal video counts, a pie chart of video location distribution, and a donut chart of abnormal event directions.
• Figure 5: Video duration and abnormal classes in VIEW360.