T$^3$Bench: Benchmarking Current Progress in Text-to-3D Generation

TL;DR

<3-5 sentence high-level summary>

Abstract

Recent methods in text-to-3D leverage powerful pretrained diffusion models to optimize NeRF. Notably, these methods are able to produce high-quality 3D scenes without training on 3D data. Due to the open-ended nature of the task, most studies evaluate their results with subjective case studies and user experiments, thereby presenting a challenge in quantitatively addressing the question: How has current progress in Text-to-3D gone so far? In this paper, we introduce T$^3$Bench, the first comprehensive text-to-3D benchmark containing diverse text prompts of three increasing complexity levels that are specially designed for 3D generation. To assess both the subjective quality and the text alignment, we propose two automatic metrics based on multi-view images produced by the 3D contents. The quality metric combines multi-view text-image scores and regional convolution to detect quality and view inconsistency. The alignment metric uses multi-view captioning and GPT-4 evaluation to measure text-3D consistency. Both metrics closely correlate with different dimensions of human judgments, providing a paradigm for efficiently evaluating text-to-3D models. The benchmarking results, shown in Fig. 1, reveal performance differences among an extensive 10 prevalent text-to-3D methods. Our analysis further highlights the common struggles for current methods on generating surroundings and multi-object scenes, as well as the bottleneck of leveraging 2D guidance for 3D generation. Our project page is available at: https://t3bench.com.

Figures (13)

• Figure 1: The average scores of 10 prevalent text-to-3D methods on T$^3$Bench, computed by the mean of quality & alignment metrics.
• Figure 2: The overview of our T$^3$Bench benchmark pipeline.
• Figure 3: Demonstration of scores at different viewpoints after multi-view capturing and regional convolution. Here, we use a level-0 icosahedron for a schematic illustration, please refer to Fig. \ref{['fig:ico']} in the supplementary material for more details.
• Figure 3: Relative quality score drop from 3D scenes without Janus problem to scenes with Janus problem.
• Figure 4: Underlying mechanism of how our multi-view quality metric reflects the Janus problem: Scores for illed-views are penalized, and regional convolution propagates this drop in local score to the global score.