Enhanced Mixture 3D CGAN for Completion and Generation of 3D Objects

TL;DR

This work tackles the challenge of generating and completing 3D objects from partial observations by integrating a Mixture-of-Experts framework with deep 3D CGANs. It introduces MoE-CGAN, featuring multiple specialized 3D CGAN generators guided by a context-aware gating network and a dynamic capacity constraint ($DCC$) to balance specialization and computational efficiency. Key innovations include geometry-aware expert specialization, Top-K context routing with sparse activation, and a geometry-consistency loss for completion, validated on ShapeNet and otolith datasets with superior CD, HD, EMD, and PRR scores and improved efficiency. The approach offers robust 3D generation and completion suitable for biological analysis and resource-constrained applications, enabling more diverse and faithful shape reconstructions from incomplete data.

Abstract

The generation and completion of 3D objects represent a transformative challenge in computer vision. Generative Adversarial Networks (GANs) have recently demonstrated strong potential in synthesizing realistic visual data. However, they often struggle to capture complex and diverse data distributions, particularly in scenarios involving incomplete inputs or significant missing regions. These challenges arise mainly from the high computational requirements and the difficulty of modeling heterogeneous and structurally intricate data, which restrict their applicability in real-world settings. Mixture of Experts (MoE) models have emerged as a promising solution to these limitations. By dynamically selecting and activating the most relevant expert sub-networks for a given input, MoEs improve both performance and efficiency. In this paper, we investigate the integration of Deep 3D Convolutional GANs (CGANs) with a MoE framework to generate high-quality 3D models and reconstruct incomplete or damaged objects. The proposed architecture incorporates multiple generators, each specialized to capture distinct modalities within the dataset. Furthermore, an auxiliary loss-free dynamic capacity constraint (DCC) mechanism is introduced to guide the selection of categorical generators, ensuring a balance between specialization, training stability, and computational efficiency, which is critical for 3D voxel processing. We evaluated the model's ability to generate and complete shapes with missing regions of varying sizes and compared its performance with state-of-the-art approaches. Both quantitative and qualitative results confirm the effectiveness of the proposed MoE-DCGAN in handling complex 3D data.

Figures (8)

• Figure 1: Our MoE-CGAN generates 3D shapes. The 3D objects shown were reconstructed using the Marching Cubes.
• Figure 2: The proposed architecture of MoE-CGAN. The gating network $GN$ takes both the latent vector $\mathbf{z}$ and partial input $\mathbf{x_p}$ (for completion tasks) to route to specialized experts.
• Figure 3: Otoliths shapes for different marine fish species.
• Figure 4: Comparisons of generation results via different methods on Chair, Airplane, and otoliths shapes.
• Figure 5: Comparisons of completion results using different methods on Chair, Airplane, and Otolith shapes. (a) Partial object, (b) Reconstructed object.