Symmetry Strikes Back: From Single-Image Symmetry Detection to 3D Generation

TL;DR

Reflect3D is introduced, a scalable, zero-shot symmetry detector capable of robust generalization to diverse and real-world scenarios and establishing a new state-of-the- art in single-image symmetry detection.

Abstract

Symmetry is a ubiquitous and fundamental property in the visual world, serving as a critical cue for perception and structure interpretation. This paper investigates the detection of 3D reflection symmetry from a single RGB image, and reveals its significant benefit on single-image 3D generation. We introduce Reflect3D, a scalable, zero-shot symmetry detector capable of robust generalization to diverse and real-world scenarios. Inspired by the success of foundation models, our method scales up symmetry detection with a transformer-based architecture. We also leverage generative priors from multi-view diffusion models to address the inherent ambiguity in single-view symmetry detection. Extensive evaluations on various data sources demonstrate that Reflect3D establishes a new state-of-the-art in single-image symmetry detection. Furthermore, we show the practical benefit of incorporating detected symmetry into single-image 3D generation pipelines through a symmetry-aware optimization process. The integration of symmetry significantly enhances the structural accuracy, cohesiveness, and visual fidelity of the reconstructed 3D geometry and textures, advancing the capabilities of 3D content creation.

Figures (6)

• Figure 1: We propose Reflect3D, a zero-shot 3D reflection symmetry detector capable of accurately detecting 3D symmetry from a single RGB image of an arbitrary object. Conditioned on the detected symmetry, we improve single-image 3D generation in both geometry and texture quality.
• Figure 2: Overview of Reflect3D, our zero-shot single-image symmetry detector. Top: Our transformer-based feed-forward symmetry detector (\ref{['sec:ff']}) predicts symmetry planes from a single RGB image. Bottom: Our multi-view symmetry enhancement pipeline (\ref{['sec:mv']}) leverages multi-view diffusion to resolve the inherent single-view ambiguity in symmetry detection. Aggregating symmetry predictions from multiple synthesized views results in more precise and comprehensive symmetry predictions.
• Figure 3: Our symmetry-aware 3D generation pipeline (\ref{['sec:sds']}). Building on DreamGaussian tang2023dreamgaussian, we integrate the detected symmetry through three steps, namely, symmetry alignment, symmetric SDS optimization, and symmetric texture refinement.
• Figure 4: Ablation studies for our single-image 3D generation pipeline. Removing each component adversely affects geometry quality, texture quality, or both.
• Figure 5: Qualitative results for our symmetry detection pipeline. Our Reflect3D achieves better generalization and precision than NeRD zhou2021nerd. Please refer to our project webpage for video results.