TexLiDAR: Automated Text Understanding for Panoramic LiDAR Data

TL;DR

The paper tackles the challenge of text-LiDAR alignment by leveraging panoramic 2D LiDAR images from the Ouster OS1. It applies the Florence 2 large model in zero-shot to perform image captioning and object detection on four 90° segments of the lidar imagery, merging outputs to form a 360° scene understanding, and uses the associated point cloud to estimate object distance and angle with $distance = \sqrt{x^2 + y^2}$ and $angle = 360 \times \frac{u_{\text{BB}} - W/2}{W}$. The contributions include bypassing 3D point-cloud processing, achieving richer captions and improved detection relative to LidarCLIP, and enabling efficient, real-time perception over full 360°. The work also outlines future multimodal fusion across ambient, reflectivity, intensity, and range lidar modalities, with broad applicability to autonomous systems and robotics.

Abstract

Efforts to connect LiDAR data with text, such as LidarCLIP, have primarily focused on embedding 3D point clouds into CLIP text-image space. However, these approaches rely on 3D point clouds, which present challenges in encoding efficiency and neural network processing. With the advent of advanced LiDAR sensors like Ouster OS1, which, in addition to 3D point clouds, produce fixed resolution depth, signal, and ambient panoramic 2D images, new opportunities emerge for LiDAR based tasks. In this work, we propose an alternative approach to connect LiDAR data with text by leveraging 2D imagery generated by the OS1 sensor instead of 3D point clouds. Using the Florence 2 large model in a zero-shot setting, we perform image captioning and object detection. Our experiments demonstrate that Florence 2 generates more informative captions and achieves superior performance in object detection tasks compared to existing methods like CLIP. By combining advanced LiDAR sensor data with a large pre-trained model, our approach provides a robust and accurate solution for challenging detection scenarios, including real-time applications requiring high accuracy and robustness.

Figures (10)

• Figure 1: The Ouster OS1 sensor offers high-resolution depth, signal, and ambient images with a 360-degree field of view, ideal for lidar-based tasks. Its perfect 1:1 spatial correspondence ensures each 2D pixel maps directly to a 3D point without resampling, reducing noise, artifacts, and computational load while enhancing the accuracy of 2D and 3D perception integrations OusterOS1.
• Figure 3: An example of our image captioning output and object detection results using 90${^\circ}$ imagery.
• Figure : Looking towards the left, the image shows a person(4.9[m], -101.7°) walking down a street in front of a building, surrounded by trees and poles. From the front perspective, the image depicts a person (6.6[m],-29.8°) walking down a street lined with trees, with cars (11[m],-17.75°) parked on the side of the road. The sky is visible in the background. As seen from the right, the image features a park with trees, plants, and a house in the background. The photo is slightly blurred, giving it a dreamy, ethereal quality. From the back viewpoint, we see a car (0.8[m],-176.5°) parked on the side of a road, surrounded by trees, poles, and a board. The sky is visible in the background.
• Figure : A house at night, surrounded by trees and a fence. The house is illuminated by the moonlight, casting a soft glow on the surrounding area, despite being partially obscured by the trees and bushes.
• Figure : Two cars parked on the side of a road, with a person riding a bicycle in the foreground. In the background, there are houses, trees, and a sky with clouds.