Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Leveraging VLM-Based Pipelines to Annotate 3D Objects

Rishabh Kabra, Loic Matthey, Alexander Lerchner, Niloy J. Mitra

TL;DR

This work tackles scalable annotation of unlabeled 3D objects via pretrained vision–language models by introducing ScoreAgg, a visually grounded, probabilistic aggregation over multi-view VLM outputs that marginalizes nuisance factors such as viewpoint. Unlike text-only LLM summaries, ScoreAgg leverages joint image–text likelihoods to produce a distribution over possible object attributes, achieving state-of-the-art type and material inference on Objaverse and enabling unsupervised measures of hallucination. The approach demonstrates strong performance against the CAP3D baseline, reduces caption blow-up, and supports downstream conditional inference when object type is provided, while scaling to 764K Objaverse objects and releasing 5M aggregated captions. A complementary unsupervised visual sensitivity metric further helps quantify when vision contributes to accuracy, supporting robust, scalable annotations for large 3D datasets and related applications in retrieval and simulation.

Abstract

Pretrained vision language models (VLMs) present an opportunity to caption unlabeled 3D objects at scale. The leading approach to summarize VLM descriptions from different views of an object (Luo et al., 2023) relies on a language model (GPT4) to produce the final output. This text-based aggregation is susceptible to hallucinations as it merges potentially contradictory descriptions. We propose an alternative algorithm to marginalize over factors such as the viewpoint that affect the VLM's response. Instead of merging text-only responses, we utilize the VLM's joint image-text likelihoods. We show our probabilistic aggregation is not only more reliable and efficient, but sets the SoTA on inferring object types with respect to human-verified labels. The aggregated annotations are also useful for conditional inference; they improve downstream predictions (e.g., of object material) when the object's type is specified as an auxiliary text-based input. Such auxiliary inputs allow ablating the contribution of visual reasoning over visionless reasoning in an unsupervised setting. With these supervised and unsupervised evaluations, we show how a VLM-based pipeline can be leveraged to produce reliable annotations for 764K objects from the Objaverse dataset.

Leveraging VLM-Based Pipelines to Annotate 3D Objects

TL;DR

This work tackles scalable annotation of unlabeled 3D objects via pretrained vision–language models by introducing ScoreAgg, a visually grounded, probabilistic aggregation over multi-view VLM outputs that marginalizes nuisance factors such as viewpoint. Unlike text-only LLM summaries, ScoreAgg leverages joint image–text likelihoods to produce a distribution over possible object attributes, achieving state-of-the-art type and material inference on Objaverse and enabling unsupervised measures of hallucination. The approach demonstrates strong performance against the CAP3D baseline, reduces caption blow-up, and supports downstream conditional inference when object type is provided, while scaling to 764K Objaverse objects and releasing 5M aggregated captions. A complementary unsupervised visual sensitivity metric further helps quantify when vision contributes to accuracy, supporting robust, scalable annotations for large 3D datasets and related applications in retrieval and simulation.

Abstract

Pretrained vision language models (VLMs) present an opportunity to caption unlabeled 3D objects at scale. The leading approach to summarize VLM descriptions from different views of an object (Luo et al., 2023) relies on a language model (GPT4) to produce the final output. This text-based aggregation is susceptible to hallucinations as it merges potentially contradictory descriptions. We propose an alternative algorithm to marginalize over factors such as the viewpoint that affect the VLM's response. Instead of merging text-only responses, we utilize the VLM's joint image-text likelihoods. We show our probabilistic aggregation is not only more reliable and efficient, but sets the SoTA on inferring object types with respect to human-verified labels. The aggregated annotations are also useful for conditional inference; they improve downstream predictions (e.g., of object material) when the object's type is specified as an auxiliary text-based input. Such auxiliary inputs allow ablating the contribution of visual reasoning over visionless reasoning in an unsupervised setting. With these supervised and unsupervised evaluations, we show how a VLM-based pipeline can be leveraged to produce reliable annotations for 764K objects from the Objaverse dataset.
Paper Structure (29 sections, 3 equations, 21 figures, 4 tables)

This paper contains 29 sections, 3 equations, 21 figures, 4 tables.

Table of Contents

  1. Introduction
  2. Background and Setting
  3. Related Work
  4. Type Annotation
  5. Visually Grounded Score-Based Aggregation
  6. Evaluation w.r.t. Human Labels
  7. An Unsupervised Measure of Hallucination
  8. Explaining ScoreAgg's performance
  9. Material Inference
  10. Evaluation w.r.t. Human Labels
  11. An Unsupervised Metric for Downstream Inference
  12. Conclusions
  13. Limitations and Future Work
  14. Outline
  15. Model Details
  16. ...and 14 more sections

Figures (21)

  • Figure 1: An inferred conceptual subgrid for 3D objects. We probe pretrained VLMs on how well they infer object properties such as type (e.g., "cup", "pot", "tower") and material (e.g., "ceramic", "wood", "plastic"). We apply a visually grounded aggregation across 2D views to avoid hallucinated outputs. Our method, ScoreAgg, produces more reliable captions than the current SoTA, CAP3D luo2023scalable, as shown in the callouts.
  • Figure 2: A. Three of eight regularly spaced views of a 3D object. Each view is accompanied by the top caption produced by two different models: BLIP-2 & PaLI-X. Captions from BLIP-2 were obtained from the competitive CAP3D baseline, whereas captions from PaLI-X were generated with accompanying scores for this work. Both models show an expected variation in responses across views. B1. To aggregate multi-view captions, CAP3D feeds them to GPT4 and prompts it for an object-level summary. The LLM is unable to reconcile captions from different views, and simply adds up the contents. B2. Our algorithm downweights unreliable responses by combining scores across views. We show its top-4 outputs.
  • Figure 3: Comparison of captions and type annotations generated from different sources/models. All results are averaged over Objaverse-LVIS. Left: the bars show text-embedding similarity scores ($\uparrow$) for the aggregate/top per-object descriptions from each source. We show an example caption/type annotation above each bar; these correspond to a fixed object shown in the upper left corner from two different views. Right: string-match metrics that assess full predicted distributions from two sources. We report top-k accuracies (whether the correct type is in the top-k predictions, $\uparrow$) as well as the soft accuracy (probability of the correct type in the output distribution, $\uparrow$).
  • Figure 4: Objects with the largest caption blow-up ratio for CAP3D. We compare their aggregate captions with ours.
  • Figure 5: Explaining ScoreAgg's performance. We apply ScoreAgg on different subsets of VLM probes ($I_v=8$ object views, $I_q=4$ VQA prompts) and VLM responses per probe (up to $J=5$). Aggregate outputs are scored on Objaverse-LVIS as before.
  • ...and 16 more figures