OmniNOCS: A unified NOCS dataset and model for 3D lifting of 2D objects

TL;DR

OmniNOCS tackles the need for scalable monocular 3D lifting across many object classes and domains by releasing a large NOCS dataset (97 categories, $380k$ images) and a transformer-based model, NOCSformer, that lifts 2D detections to NOCS maps, instance masks, and metric 3D pose without class-specific heads. The approach leverages pre-trained ViT backbones with a PnP-based head to recover 3D orientation and centroid, enabling 3D oriented boxes and object-shaped point clouds from a single image. The authors also propose the OmniNOCS benchmark and demonstrate strong cross-dataset generalization, including zero-shot transfer to NOCS-Real275 and competitive outdoor localization on nuScenes, while maintaining canonical orientations and explicit segmentation. Overall, this work provides a practical, scalable baseline for large-scale monocular 3D pose and shape estimation across diverse object vocabularies.

Abstract

We propose OmniNOCS, a large-scale monocular dataset with 3D Normalized Object Coordinate Space (NOCS) maps, object masks, and 3D bounding box annotations for indoor and outdoor scenes. OmniNOCS has 20 times more object classes and 200 times more instances than existing NOCS datasets (NOCS-Real275, Wild6D). We use OmniNOCS to train a novel, transformer-based monocular NOCS prediction model (NOCSformer) that can predict accurate NOCS, instance masks and poses from 2D object detections across diverse classes. It is the first NOCS model that can generalize to a broad range of classes when prompted with 2D boxes. We evaluate our model on the task of 3D oriented bounding box prediction, where it achieves comparable results to state-of-the-art 3D detection methods such as Cube R-CNN. Unlike other 3D detection methods, our model also provides detailed and accurate 3D object shape and segmentation. We propose a novel benchmark for the task of NOCS prediction based on OmniNOCS, which we hope will serve as a useful baseline for future work in this area. Our dataset and code will be at the project website: https://omninocs.github.io.

Figures (13)

• Figure 1: We introduce OmniNOCS, a large-scale dataset with Normalized Object Coordinates (NOCS), instance masks, and 3D box annotations for objects across several classes, domains, and cameras. We also propose NOCSformer, a model trained on OmniNOCS that lifts each 2D object bounding box in an image to its corresponding 3D oriented box (pose) and 3D pointcloud (shape).
• Figure 2: Preparation of the OmniNOCS dataset: First, we ensure that object orientations are canonical across different all subsets of OmniNOCS. Next, we re-compute depth for datasets where depth is not available or is too noisy. Finally, we annotate objects with (pseudo) instance mask labels where ground truth masks are not available.
• Figure 3: Architecture for NOCSformer: We use a transformer backbone to extract features from the input image, pool them using the 2D box RoIs, and feed the per-RoI features to the NOCS and size heads. Our novel NOCS head jointly predicts the NOCS and instance mask for the RoI. Our learned PnP head for pose estimation uses the predicted NOCS and instance mask to predict the projected 3D coordinate and 3D rotation of the object.
• Figure 4: Example results of our single unified NOCSformer model across various datasets and object classes. The left column shows input images and query 2D bounding boxes. The center column shows the NOCS+instance maps predicted by NOCSformer along with the estimated 3D pose overlaid on the input image. The NOCS can be used with the 3D boxes and object size to lift the objects to a 3D pointcloud, which is shown in the right column. Last row contains two examples.
• Figure 5: Generalization across datasets: NOCSformer can generalize to new datasets that present new camera models and object domains. We show this zero-shot capability of NOCSformer (bottom row) on the NOCS-Real275 test set without training on NOCS-Real275 dataset. The predictions from a NOCS baseline Wang_2019_CVPR trained explicitly on the NOCS-Real275 dataset are shown in the top row.