View Selection for 3D Captioning via Diffusion Ranking

TL;DR

This work tackles hallucinations in 3D captioning arising from accidental render views in Cap3D by introducing DiffuRank, a view-ranking method that uses a pre-trained text-to-3D diffusion prior to guide captioning. By selecting top-ranked views and employing GPT4-Vision, the approach improves caption accuracy and detail, enabling correction of about 200k Cap3D captions and expanding the dataset to roughly 1.5M captions across Objaverse and Objaverse-XL. The method is also extended to the 2D domain for Visual Question Answering, where DiffuRank with a text-to-2D diffusion model outperforms CLIP. The authors release the expanded Cap3D dataset under ODC-By, demonstrate improved downstream 3D-text model finetuning, and show DiffuRank’s versatility across modalities.

Abstract

Scalable annotation approaches are crucial for constructing extensive 3D-text datasets, facilitating a broader range of applications. However, existing methods sometimes lead to the generation of hallucinated captions, compromising caption quality. This paper explores the issue of hallucination in 3D object captioning, with a focus on Cap3D method, which renders 3D objects into 2D views for captioning using pre-trained models. We pinpoint a major challenge: certain rendered views of 3D objects are atypical, deviating from the training data of standard image captioning models and causing hallucinations. To tackle this, we present DiffuRank, a method that leverages a pre-trained text-to-3D model to assess the alignment between 3D objects and their 2D rendered views, where the view with high alignment closely represent the object's characteristics. By ranking all rendered views and feeding the top-ranked ones into GPT4-Vision, we enhance the accuracy and detail of captions, enabling the correction of 200k captions in the Cap3D dataset and extending it to 1 million captions across Objaverse and Objaverse-XL datasets. Additionally, we showcase the adaptability of DiffuRank by applying it to pre-trained text-to-image models for a Visual Question Answering task, where it outperforms the CLIP model.

Figures (9)

• Figure 1: DiffuRank enhances caption accuracy and reduces hallucinations by prioritizing key rendered views (green box), contrasting the accidental views freeman1994generic (red box) that cause errors. Surprisingly, using fewer views (6 vs. 28) not only saves computational resources but also may yield more accurate and detailed outcomes (the middle example) by countering the uncertainty caused by excessive views.
• Figure 2: The left row features the top-6 views as ranked by DiffuRank, while the right row displays the bottom-6. Comparative analysis shows that the top-6 views generally uncover more characteristics of the object compared to the bottom-6. This finding underscores DiffuRank's capability to identify views that more accurately represent the features of the 3D object. More randomly sampled results are included in Appendix B.5.
• Figure 3: Methods overview. Both Cap3D and our method render input 3D objects into multiple views for caption generation (green steps). However, while Cap3D consolidates these captions into a final description (blue steps), our method employs a pre-trained text-to-3D diffusion model to identify views that better match the input object's characteristics. These selected views are then processed by a Vision-Language Model (VLM) for captioning (orange steps).
• Figure 3: Accuracy comparison among various VLMs, CLIP, and our method.
• Figure 4: We utilized both grey background + ray-tracing render engine (left images) and transparent background + real-time render engine (right images) for rendering, discovering that the effectiveness of each varies. We noticed DiffuRank can select the views with the appropriate rendering that highlight object features.