LiDARDraft: Generating LiDAR Point Cloud from Versatile Inputs

TL;DR

LiDARDraft tackles the challenge of producing realistic and controllable LiDAR point clouds for autonomous-driving simulation by unifying text, image, and point-cloud inputs into a 3D layout representation that feeds a diffusion-based generator. A raycast-derived, pixel-aligned range-image control signals—derived from semantic primitives—guide a ControlNet-tuned diffusion model while an unconditional diffusion backbone remains fixed for efficiency. The approach demonstrates strong cross-modal controllability and high-quality outputs across KITTI-360, nuScenes, and SemanticKITTI, outperforming state-of-the-art baselines on standard metrics and enabling scalable “simulation from scratch” through text, image, or sketch prompts. Ablation confirms the necessity of components such as the layout projection, semantic segmentation, depth cues, and the ControlNet fine-tuning for achieving the reported gains, highlighting practical paths to further speedups and multi-domain applicability.

Abstract

Generating realistic and diverse LiDAR point clouds is crucial for autonomous driving simulation. Although previous methods achieve LiDAR point cloud generation from user inputs, they struggle to attain high-quality results while enabling versatile controllability, due to the imbalance between the complex distribution of LiDAR point clouds and the simple control signals. To address the limitation, we propose LiDARDraft, which utilizes the 3D layout to build a bridge between versatile conditional signals and LiDAR point clouds. The 3D layout can be trivially generated from various user inputs such as textual descriptions and images. Specifically, we represent text, images, and point clouds as unified 3D layouts, which are further transformed into semantic and depth control signals. Then, we employ a rangemap-based ControlNet to guide LiDAR point cloud generation. This pixel-level alignment approach demonstrates excellent performance in controllable LiDAR point clouds generation, enabling "simulation from scratch", allowing self-driving environments to be created from arbitrary textual descriptions, images and sketches.

Figures (8)

• Figure 1: Overview of LiDARDraft. Various inputs are unified into layout representations and projected into range images using RayCasting, which are then fed into ControlNet to guide unconditional LiDAR point cloud generation. Image first undergoes semantic segmentation and depth estimation to form a pseudo-point cloud and then clustered to create the layout. Point cloud is semantically segmented and then clustered. Text can generate layout using a Large Language Model. Moreover, we can easily modify the layout to edit the scene.
• Figure 2: Layout to LiDAR Point Cloud results. The white boxes indicate vehicle locations in the layout, and the blue boxes show the generated vehicle positions. The rightmost column presents samples of the remove-car manipulation.
• Figure 3: Image to LiDAR Point Cloud generation. The orange boxes mark the area covered by the input image.In the left column, the vehicles in the input image are accurately sampled by both our method and LiDARDiffusion. However, the LiDARDiffusion sampling results contain multiple interfering vehicles. In the right column, the input image shows an empty road, and our method generates a point cloud along the straight path. However, LiDARDiffusion erroneously generates vehicles.
• Figure 4: Text (left half) / Point Cloud (right half) to LiDAR Point Cloud generation.For text: an automated driving scene layout description is generated with GPT and used to sample the point cloud. LiDARDraft accurately samples the point cloud consistent with the text description, recognizing road types, vehicle counts, and their relative positions. The blue and pink boxes highlight specific details of the sampled point cloud. For point cloud: given a single-frame point cloud as input, LiDARDraft can sample multiple point clouds with consistent layouts and ensure layout consistency while maintaining diversity. The blue and pink boxes highlight the local details of the sampled point clouds.
• Figure 5: Qualitative Results on the SemanticKITTI Dataset. We select three consecutive LiDAR frames as the conditioning input and observe that the generated point clouds remain temporally consistent, geometrically plausible, and visually realistic across the sequence.