MAROON: A Dataset for the Joint Characterization of Near-Field High-Resolution Radio-Frequency and Optical Depth Imaging Techniques

TL;DR

MAROON presents the first publicly available near-field, multimodal dataset that jointly characterizes optical depth sensors and high-resolution RF imaging radar. By mutually calibrating three optical depth imagers and a MIMO RF ToF radar against a ground-truth MVS system, the work offers a comprehensive evaluation framework quantifying depth deviations and RF signal responses across object materials, geometries, and distances. Key findings show systematic ToF scattering in partially transmissive media for both NIR and RF modalities, and reveal that RF reconstructions are often sparser and more geometry-dependent than optical counterparts. The dataset enables material characterization via differentiable ray tracing and supports multimodal reconstruction benchmarking, with potential impact on realistic radar simulation and development of robust multimodal depth-sensing pipelines for close-range applications.

Abstract

Utilizing the complementary strengths of wavelength-specific range or depth sensors is crucial for robust computer-assisted tasks such as autonomous driving. Despite this, there is still little research done at the intersection of optical depth sensors and radars operating close range, where the target is decimeters away from the sensors. Together with a growing interest in high-resolution imaging radars operating in the near field, the question arises how these sensors behave in comparison to their traditional optical counterparts. In this work, we take on the unique challenge of jointly characterizing depth imagers from both, the optical and radio-frequency domain using a multimodal spatial calibration. We collect data from four depth imagers, with three optical sensors of varying operation principle and an imaging radar. We provide a comprehensive evaluation of their depth measurements with respect to distinct object materials, geometries, and object-to-sensor distances. Specifically, we reveal scattering effects of partially transmissive materials and investigate the response of radio-frequency signals. All object measurements will be made public in form of a multimodal dataset, called MAROON.

Figures (19)

• Figure 1: Example data of the Plunger object from the MAROON dataset. In the upper left, all reconstructions are spatially aligned with respect to the RF ToF coordinate system. The RF ToF colorscale encodes the normalized reconstruction confidence (cf. \ref{['sec:mimo_fscw']}).
• Figure 2: Visualization of the effects caused by limited spatial resolution for multiple point targets. Optical sensors (left) have similar definitions as far-field RF ToF sensors and divide spatial resolution into depth, $\delta_r$, and pixel resolution, $\delta_{h,v}$. Contrary to that, near-field imaging radars (right) refer to range $\delta_z$ and cross-range $\delta_{x,y}$ resolution. We assume $\delta_z \approx \delta_r$ and $\delta_{x,y} \approx \delta_{h,v}$ for the sensor center, yet emphasize the conceptual difference between range and depth.
• Figure 3: Overview of the two depth sensing categories considered in this work. Spatially resolved methods compute the depth from disparity in the pixel positions. Time-resolved methods measure the depth through the round-trip propagation time of the received continuous wave (CW) signal. The types of wave forms utilized in our experiments are amplitude-modulated continuous wave (AMCW) and frequency-stepped continuous wave (FSCW).
• Figure 4: On the left, object material, geometry (median surface incidence angle), and size (relative surface area) are put in relation to received signal response (mean signal magnitude, top row) and mean depth deviation (bottom row). On the right, both quantities are directly compared to each other. M easurements, where large objects appear outside the radar's antenna aperture , are highlighted in gray regions, as they exhibit higher depth deviations compared to the ground-truth reconstructions, which may extend beyond this aperture; this is attributed to the comparably small field of view and the surface reflection characteristics with respect to radio waves (see supp. mat.). The results are discussed in \ref{['sec:discussion_radar_signal']}.
• Figure 5: For selected objects, we show the reconstructed point clouds (left) next to their deviation from to the MVS reconstruction (right). The signed depth deviation P* is given for each pixel $(u,v)$ in centimeters. All measurements in the domain $\boldsymbol{M}^{+}(u,v)$ are projected onto the GT reconstruction and mapped to color using a combination of a symmetrical logarithmic scale and linear mapping between $[-0.5,0.5]$ centimeters . T he mean deviation of P* is quantified in centimeters below each sensor measurement.