OpenBox: Annotate Any Bounding Boxes in 3D

TL;DR

OpenBox introduces a two-stage automatic 3D bounding box annotation pipeline that leverages 2D vision foundation models to annotate open-vocabulary object categories in 3D scenes. The method first aligns 2D instance cues to 3D point clouds and then adaptively generates bounding boxes by classifying instances into static rigid, dynamic rigid, and deformable types with scene- and geometry-guided refinements. Key contributions include context-aware and surface-aware point refinement, PP-score-based dynamic handling, SDF-based surface filtering, and 3D-2D IoU-based box adjustment, all without self-training. Empirical evaluation on Waymo Open Dataset, Lyft, and nuScenes demonstrates superior annotation quality and efficiency compared with strong baselines, highlighting the practicality of open-vocabulary 3D annotation for autonomous driving.

Abstract

Unsupervised and open-vocabulary 3D object detection has recently gained attention, particularly in autonomous driving, where reducing annotation costs and recognizing unseen objects are critical for both safety and scalability. However, most existing approaches uniformly annotate 3D bounding boxes, ignore objects' physical states, and require multiple self-training iterations for annotation refinement, resulting in suboptimal quality and substantial computational overhead. To address these challenges, we propose OpenBox, a two-stage automatic annotation pipeline that leverages a 2D vision foundation model. In the first stage, OpenBox associates instance-level cues from 2D images processed by a vision foundation model with the corresponding 3D point clouds via cross-modal instance alignment. In the second stage, it categorizes instances by rigidity and motion state, then generates adaptive bounding boxes with class-specific size statistics. As a result, OpenBox produces high-quality 3D bounding box annotations without requiring self-training. Experiments on the Waymo Open Dataset, the Lyft Level 5 Perception dataset, and the nuScenes dataset demonstrate improved accuracy and efficiency over baselines.

Figures (9)

• Figure 1: We introduce OpenBox, which utilizes a 2D vision foundation model to annotate 3D bounding boxes automatically. It annotates instances of vehicles, pedestrians, and cyclists. We demonstrate it with Waymo Open Dataset waymo. Best viewed in color and zoomed in.
• Figure 2: Pipeline Overview of OpenBox. With time-synchronized, unlabeled images and LiDAR scans, cross-modal instance alignment (Sec \ref{['sec:methodology:featureextraction']}) associates 2D instance cues with corresponding point clouds. Adaptive 3D bounding box generation (Sec \ref{['sec:methodology:adaptiveboxgen']}) independently chooses the most suitable fitting strategy for each instance, yielding high-quality 3D bounding boxes.
• Figure 3: Cross-modal Instance Alignment. To obtain a refined point cloud, the pipeline generates two complementary point cloud clusters. The LiDAR (upper) branch removes ground points and applies HDBSCAN hdbscan to produce coarse 3D clusters. The image-LiDAR (lower) branch uses Grounding DINO groundingdino followed by SAM2 samv2 to generate 2D instance masks. This information is unprojected to a point cloud. Context-aware refinement fuses the two proposals, discarding noisy points and incorporating adjacent points from the point cloud cluster, yielding a refined per-object point cloud.
• Figure 4: Context-aware Refinement. (a) Reference image. (b) Point cloud clusters $\mathcal{R}$ after using HDBSCAN hdbscan on ground-removed LiDAR point cloud. (c) Noisy instance-level point clouds $\mathcal{F}$. (d) Result of the Context-aware refinement $\mathcal{F}_{\text{ref}}$.
• Figure 5: Handling Static & Rigid Instances. (a) We filter noisy points in the aggregated static point cloud via vertex-level voting on the reconstructed surface, producing $\mathcal{F}^{S,(2)}_{\text{ref}}$ and $\mathbf{S}_{\text{ref}}$. (b) We then adjust the bounding box using surface normals and statistical priors, and select the final box based on 2D IoU with projected boxes and Grounding DINO groundingdino boxes.