Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

mmWave Radar-Based Non-Line-of-Sight Pedestrian Localization at T-Junctions Utilizing Road Layout Extraction via Camera

Byeonggyu Park, Hee-Yeun Kim, Byonghyok Choi, Hansang Cho, Byungkwan Kim, Soomok Lee, Mingu Jeon, Seong-Woo Kim

TL;DR

This work tackles NLoS pedestrian localization in urban intersections by fusing mmWave radar 2D point clouds with camera-derived road layouts. The core approach interprets sparse, distorted radar PCD using camera-derived road layout, employing edge extraction, reflector learning, and ray tracing to reconstruct the scene and localize pedestrians. The method is validated on a real-vehicle radar-camera setup with outdoor NLoS data, showing sub-meter Absolute Errors and improved angular localization over radar-only baselines. The work advances practical NLoS perception for autonomous driving by enabling reliable pedestrian localization in occluded regions at complex junctions.

Abstract

Pedestrians Localization in Non-Line-of-Sight (NLoS) regions within urban environments poses a significant challenge for autonomous driving systems. While mmWave radar has demonstrated potential for detecting objects in such scenarios, the 2D radar point cloud (PCD) data is susceptible to distortions caused by multipath reflections, making accurate spatial inference difficult. Additionally, although camera images provide high-resolution visual information, they lack depth perception and cannot directly observe objects in NLoS regions. In this paper, we propose a novel framework that interprets radar PCD through road layout inferred from camera for localization of NLoS pedestrians. The proposed method leverages visual information from the camera to interpret 2D radar PCD, enabling spatial scene reconstruction. The effectiveness of the proposed approach is validated through experiments conducted using a radar-camera system mounted on a real vehicle. The localization performance is evaluated using a dataset collected in outdoor NLoS driving environments, demonstrating the practical applicability of the method.

mmWave Radar-Based Non-Line-of-Sight Pedestrian Localization at T-Junctions Utilizing Road Layout Extraction via Camera

TL;DR

This work tackles NLoS pedestrian localization in urban intersections by fusing mmWave radar 2D point clouds with camera-derived road layouts. The core approach interprets sparse, distorted radar PCD using camera-derived road layout, employing edge extraction, reflector learning, and ray tracing to reconstruct the scene and localize pedestrians. The method is validated on a real-vehicle radar-camera setup with outdoor NLoS data, showing sub-meter Absolute Errors and improved angular localization over radar-only baselines. The work advances practical NLoS perception for autonomous driving by enabling reliable pedestrian localization in occluded regions at complex junctions.

Abstract

Pedestrians Localization in Non-Line-of-Sight (NLoS) regions within urban environments poses a significant challenge for autonomous driving systems. While mmWave radar has demonstrated potential for detecting objects in such scenarios, the 2D radar point cloud (PCD) data is susceptible to distortions caused by multipath reflections, making accurate spatial inference difficult. Additionally, although camera images provide high-resolution visual information, they lack depth perception and cannot directly observe objects in NLoS regions. In this paper, we propose a novel framework that interprets radar PCD through road layout inferred from camera for localization of NLoS pedestrians. The proposed method leverages visual information from the camera to interpret 2D radar PCD, enabling spatial scene reconstruction. The effectiveness of the proposed approach is validated through experiments conducted using a radar-camera system mounted on a real vehicle. The localization performance is evaluated using a dataset collected in outdoor NLoS driving environments, demonstrating the practical applicability of the method.

Paper Structure

This paper contains 24 sections, 19 equations, 4 figures, 3 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Related Works
  3. Camera-based road layout estimation model
  4. mmWave radar based NLoS object detection
  5. Problem Definition
  6. NLoS Pedestrian Localization Pipeline through Fusion of Image & PCD
  7. Inference of spatial configuration
  8. Inference of road layout
  9. Extract edge points in $I_{layout}$
  10. Alignment of $W$ & radar PCD
  11. Initial interpretation of $S$ using $W_{\text{aligned}}$
  12. Ray tracing for reflect static points
  13. Inference final spatial configuration
  14. Localization of NLoS pedestrian
  15. Ray tracing of dynamic points
  16. ...and 9 more sections

Figures (4)

  • Figure 1: Illustration of the results using the proposed method in an intersection composed of backroads. (a) Using only LoS sensors, it is not possible to detect objects in the NLoS region, (b) The proposed method allows for estimating the position of NLoS objects.
  • Figure 2: The overall framework for NLoS pedestrian localization and the results of each algorithm block.
  • Figure 3: Comparison of walls estimated Using LiDAR PCD, aligned camera points, and radar PCD. (Left) The original radar static points are sparse and contain noise. (Right) The walls estimated using the proposed method are closely aligned with the LiDAR points, showcasing the effectiveness of the approach.
  • Figure 4: Qualitative results in each scenarios. The figure is organized into four scenarios, displayed from top to bottom. Each scenario is evaluated over time, with frames progressing left to right. Additionally, the distance between each AR mark is 4m and speed of pedestrians is approximately 1.5 $m/s$.