Real-Time Glass Detection and Reprojection using Sensor Fusion Onboard Aerial Robots

TL;DR

This work tackles the challenge of safely navigating environments with transparent obstacles by proposing a lightweight, onboard framework that fuses ToF depth data with an ultrasonic reading. A custom 2D convolution-based speckle detector identifies specular reflections on glass, and the corresponding depth is propagated into empty depth regions to produce a fused depth map that renders glass as a tangible plane for planning. Key contributions include the integration of a low-SWaP sensor suite, efficient speckle-detection kernels with temporal tracking, and a robust transparent-plane reprojection method validated through extensive onboard experiments, achieving real-time performance on CPU-only hardware. The results demonstrate high precision/recall in controlled and real-world settings and show advantages over heavier RGB-D-based baselines, enabling safer autonomous navigation in glass-filled indoor spaces. The work also provides open-source datasets and sets the stage for extending to more complex glass configurations and other transparent materials.

Abstract

Autonomous aerial robots are increasingly being deployed in real-world scenarios, where transparent obstacles present significant challenges to reliable navigation and mapping. These materials pose a unique problem for traditional perception systems because they lack discernible features and can cause conventional depth sensors to fail, leading to inaccurate maps and potential collisions. To ensure safe navigation, robots must be able to accurately detect and map these transparent obstacles. Existing methods often rely on large, expensive sensors or algorithms that impose high computational burdens, making them unsuitable for low Size, Weight, and Power (SWaP) robots. In this work, we propose a novel and computationally efficient framework for detecting and mapping transparent obstacles onboard a sub-300g quadrotor. Our method fuses data from a Time-of-Flight (ToF) camera and an ultrasonic sensor with a custom, lightweight 2D convolution model. This specialized approach accurately detects specular reflections and propagates their depth into corresponding empty regions of the depth map, effectively rendering transparent obstacles visible. The entire pipeline operates in real-time, utilizing only a small fraction of a CPU core on an embedded processor. We validate our system through a series of experiments in both controlled and real-world environments, demonstrating the utility of our method through experiments where the robot maps indoor environments containing glass. Our work is, to our knowledge, the first of its kind to demonstrate a real-time, onboard transparent obstacle mapping system on a low-SWaP quadrotor using only the CPU.

Figures (8)

• Figure 1: The experimental setup and algorithmic overview of our framework. Our methodology integrates data from an ultrasonic sensor and a ToF camera to detect specular reflections (speckles) on glass surfaces and segment the depth image based on empty space. This information is then used to convert the raw depth map into a fused depth map by reprojecting the speckle depth into its respective segmented region, effectively making glass visible for safe navigation and mapping.
• Figure 2: Our robot is a ModalAI Starling 2 platform with a PMD Flexx 2 VGA Time-of-Flight sensor and a MaxSonar MB1020 Ultrasonic sensor.
• Figure 3: An Illustration of speckle formation. (a) Depicts three light rays from the ToF sensor approaching a tilted glass surface. (b) The light rays reflect, and the one perpendicular to the glass surface reflects back to the ToF sensor. (c) This returning ray creates the speckle, from which we can extract both the depth and the normal of the glass pane.
• Figure 4: Examples of specular reflections (speckles) in a Time-of-Flight depth image. (a) A close-up view of a single, isolated speckle. (b) The same speckle within the context of a glass pane at a distance of 2 meters. (c) A different speckle and its environment at a distance greater than 2 meters.
• Figure 5: Qualitative results for Speckle Detection and Glass Segmentation. The figure illustrates the performance of our algorithm (b) in segmenting transparent planes from both clear and cluttered environments. Rows 1 and 2 show results from head-on and angled approaches to a clear glass pane, while rows 3 and 4 show results from similar approaches with background objects.