LiDAR-Guided Monocular 3D Object Detection for Long-Range Railway Monitoring

TL;DR

Problem: achieving robust long-range 3D object detection for autonomous trains using only monocular images. Approach: a LiDAR-guided training pipeline (Monocular Faraway-Frustum, MFF) with a depth estimation module, a 2.5D detection head, a frustum-based decision block, and dedicated long- and short-range 3D heads. Contributions: monocular-only railway perception with LiDAR-informed depth guidance, a four-module architecture, and detailed evaluation on OSDaR23 showing detections up to 250 m and competitive baselines. Findings/Significance: demonstrates viable long-range perception for railway automation while identifying depth accuracy and data distribution as key limiting factors, pointing to future work in real-time deployment, end-to-end training, and synthetic data augmentation.

Abstract

Railway systems, particularly in Germany, require high levels of automation to address legacy infrastructure challenges and increase train traffic safely. A key component of automation is robust long-range perception, essential for early hazard detection, such as obstacles at level crossings or pedestrians on tracks. Unlike automotive systems with braking distances of ~70 meters, trains require perception ranges exceeding 1 km. This paper presents an deep-learning-based approach for long-range 3D object detection tailored for autonomous trains. The method relies solely on monocular images, inspired by the Faraway-Frustum approach, and incorporates LiDAR data during training to improve depth estimation. The proposed pipeline consists of four key modules: (1) a modified YOLOv9 for 2.5D object detection, (2) a depth estimation network, and (3-4) dedicated short- and long-range 3D detection heads. Evaluations on the OSDaR23 dataset demonstrate the effectiveness of the approach in detecting objects up to 250 meters. Results highlight its potential for railway automation and outline areas for future improvement.

Figures (6)

• Figure 1: MFF Pipeline. Top: Pipeline during training, when LiDAR point clouds are used to guide depth estimation. Bottom: Pipeline during testing time, in this case the method works solely using monocular images.
• Figure 2: Number of point cloud points within the 3D bounding box of an object, plotted against the x-axis distance from the camera to that same object. Top: Training split, Middle: Validation split, Bottom: Test split.
• Figure 3: Depth estimation module prediction example (relative depth). Left: Ground truth, Center: Input, and Right: Prediction.
• Figure 4: Depth estimation module prediction example (absolute depth). Left: Ground truth, Center: Input, and Right: Prediction.
• Figure 5: Heatmap exemplifying the error distribution in the predicted depth maps. Error increases considerably in very distant regions.