SeMv-3D: Towards Concurrency of Semantic and Multi-view Consistency in General Text-to-3D Generation

TL;DR

This paper tackles general text-to-3D (GT23D) generation, focusing on simultaneously achieving semantic alignment with text and multi-view consistency across views. It introduces SeMv-3D, a two-stage framework consisting of Triplane Prior Learning (TPL) to build a geometrically coherent triplane prior with Background Elimination and Orthogonal Attention, and Prior-based Semantic Aligning in Triplanes (SAT) to align semantics via Cross-Attention and Orthogonal Attention for any-view synthesis. The approach yields state-of-the-art multi-view consistency while maintaining competitive semantic fidelity, validated on the Objaverse dataset with comprehensive ablations demonstrating the contributions of BE, TL, OA, and the SAT components. These results highlight SeMv-3D as a strong, unified foundation for open-domain 3D generation and potentially future 4D extensions.

Abstract

General Text-to-3D (GT23D) generation is crucial for creating diverse 3D content across objects and scenes, yet it faces two key challenges: 1) ensuring semantic consistency between input text and generated 3D models, and 2) maintaining multi-view consistency across different perspectives within 3D. Existing approaches typically address only one of these challenges, often leading to suboptimal results in semantic fidelity and structural coherence. To overcome these limitations, we propose SeMv-3D, a novel framework that jointly enhances semantic alignment and multi-view consistency in GT23D generation. At its core, we introduce Triplane Prior Learning (TPL), which effectively learns triplane priors by capturing spatial correspondences across three orthogonal planes using a dedicated Orthogonal Attention mechanism, thereby ensuring geometric consistency across viewpoints. Additionally, we present Prior-based Semantic Aligning in Triplanes (SAT), which enables consistent any-view synthesis by leveraging attention-based feature alignment to reinforce the correspondence between textual semantics and triplane representations. Extensive experiments demonstrate that our method sets a new state-of-the-art in multi-view consistency, while maintaining competitive performance in semantic consistency compared to methods focused solely on semantic alignment. These results emphasize the remarkable ability of our approach to effectively balance and excel in both dimensions, establishing a new benchmark in the field.

Figures (7)

• Figure 1: Comparison with SOTA baselines and our SeMv-3D. The two mainstream lines of general text-to-3d: a) Prior-based methods and b) Fine-tuning-based methods struggle with two core challenges: semantic inconsistency and multi-view inconsistency, respectively. Our SeMv-3D c) can ensure the concurrency of semantic and multi-view consistency.
• Figure 2: The overall framework of SeMv-3D. SeMv-3D consists of two components: 1) Triplane Prior Learning (TPL) that learns a triplane prior to capture consistent 3D visual details and 2) Prior-based Semantic Aligning in Triplanes (SAT) that enhances the alignment between the semantic with 3D content and enables single-step generation of arbitrary views. Here, Orthogonal Attention (OA) focuses on the orthogonal correspondences within the triplane, maintaining triplane consistency.
• Figure 3: Qualitative Comparison with Prior-based GT23D Methods. It shows our method maintains better semantic and multi-view consistency than Prior-based Methods. More results are presented in the Suppl. C
• Figure 4: Qualitative Comparison with Fine-tuning-based GT23D Methods. It demonstrates that our method achieves superior multi-view consistency compared to fine-tuning-based methods while maintaining comparable semantic consistency. More results are presented in the Suppl. C.
• Figure 5: Multi-view Consistency Challenges in Fine-tuning-based Methods: Generating more than four views leads to severe geometric and texture inconsistencies. In contrast, our method demonstrates superior multi-view consistency compared to existing fine-tuning-based methods in any-view synthesis.