MITO: A Millimeter-Wave Dataset and Simulator for Non-Line-of-Sight Perception

TL;DR

MITO addresses the core problem that optical perception fails under occlusion by introducing a large mmWave LOS/NLOS dataset and an open-source SAR-based simulator. The approach combines real captures with two mmWave frequencies and an RGB-D sensor, plus a simulator that renders synthetic mmWave images from 3D meshes to support training and benchmarking. The key contributions are over 24 million real mmWave frames producing 550 high-resolution SAR images across 76 YCB objects, ground-truth segmentation, and a multi-reflection simulator that merges specular and edge reflections; benchmarks for NLOS segmentation and Sim2Real shape classification are demonstrated. The work significantly advances mmWave perception for robotics and logistics by providing a scalable, multimodal dataset, realistic simulation tools, and practical baselines for NLOS tasks.

Abstract

The ability to observe the world is fundamental to reasoning and making informed decisions on how to interact with the environment. However, optical perception can often be disrupted due to common occurrences, such as occlusions, which can pose challenges to existing vision systems. We present MITO, the first millimeter-wave (mmWave) dataset of diverse, everyday objects, collected using a UR5 robotic arm with two mmWave radars operating at different frequencies and an RGB-D camera. Unlike visible light, mmWave signals can penetrate common occlusions (e.g., cardboard boxes, fabric, plastic) but each mmWave frame has much lower resolution than typical cameras. To capture higher-resolution mmWave images, we leverage the robot's mobility and fuse frames over the synthesized aperture. MITO captures over 24 million mmWave frames and uses them to generate 550 high-resolution mmWave (synthetic aperture) images in line-of-sight and non-light-of-sight (NLOS), as well as RGB-D images, segmentation masks, and raw mmWave signals, taken from 76 different objects. We develop an open-source simulation tool that can be used to generate synthetic mmWave images for any 3D triangle mesh. Finally, we demonstrate the utility of our dataset and simulator for enabling broader NLOS perception by developing benchmarks for NLOS segmentation and classification.

Figures (8)

• Figure 1: MITO.We use a robotic arm to move mmWave radars throughout the environment. While an RGB-D camera's output cannot see inside the box, we produce high-resolution non-line-of-sight mmWave images.
• Figure 2: mmWave Imaginga) mmWave radars estimate range and angle-of-arrival to produce b) reflection maps.
• Figure 3: Diffuse and Specular Reflections.The frequency of a signal changes the type of reflections it experiences for the same surface.
• Figure 4: Sample Images.A number of example mmWave images within MITO.
• Figure 5: Multi-Spectral Imaging.a-b) show 77 GHz & 24 GHz images for a padlock in LOS. c-d) show the images in NLOS.