Unleashing Network Potentials for Semantic Scene Completion

TL;DR

The proposed AMMNet introduces a cross-modal modulation enabling the interdependence of gradient flows between modalities, and a customized adversarial training scheme leveraging dynamic gradient competition, providing a promising direction for improving the effectiveness and generalization of SSC methods.

Abstract

Semantic scene completion (SSC) aims to predict complete 3D voxel occupancy and semantics from a single-view RGB-D image, and recent SSC methods commonly adopt multi-modal inputs. However, our investigation reveals two limitations: ineffective feature learning from single modalities and overfitting to limited datasets. To address these issues, this paper proposes a novel SSC framework - Adversarial Modality Modulation Network (AMMNet) - with a fresh perspective of optimizing gradient updates. The proposed AMMNet introduces two core modules: a cross-modal modulation enabling the interdependence of gradient flows between modalities, and a customized adversarial training scheme leveraging dynamic gradient competition. Specifically, the cross-modal modulation adaptively re-calibrates the features to better excite representation potentials from each single modality. The adversarial training employs a minimax game of evolving gradients, with customized guidance to strengthen the generator's perception of visual fidelity from both geometric completeness and semantic correctness. Extensive experimental results demonstrate that AMMNet outperforms state-of-the-art SSC methods by a large margin, providing a promising direction for improving the effectiveness and generalization of SSC methods.

Figures (7)

• Figure 1: Comparisons of encoder representation power. For the multi-modal training in (b), the informative representations in RGB and TSDF are not fully unleashed compared to learning them individually in (a). The proposed Adversarial Modality Modulation Network (AMMNet) in (c) enables a more thorough unleashing of potentials via "cross-modal modulation $\mathcal{M}$" and "adversarial training $\mathcal{L}_{(D,G)}$".
• Figure 2: Two key observations on multi-modal SSC models. (a) Performance drops of "multi-modal encoders" compared to single-modal counterparts, validating insufficient unleashing of modalities in joint training. "Our method" demonstrates significantly enhanced encoder capabilities. (b)"Diverging training/validation curves" of baseline and the variant AMMNet, indicating overfitting issues. Under the adversarial training scheme $\mathcal{L}_{(D,G)}$, our model alleviates overfitting and achieves "steadily increasing performance".
• Figure 3: The overall framework of our AMMNet. It consists of three components: an image encoder for RGB input, a TSDF encoder for TSDF input, and a decoder for final prediction. It has two novel modules: cross-modal modulations after the encoders and decoder to recalibrate features, and a discriminator that distinguishes real/fake voxels to mitigate overfitting issues. The conv($k$, $s$)/Deconv($k$, $s$) denotes 3D conv/deconv layer with kernel size $k$ and stride $s$, and DDR($d$, $s$) denotes DDR layer li2019rgbd with dilation $d$ and stride $s$.
• Figure 4: Qualitative comparison on challenging indoor scenes from the test set of NYU silberman2012indoor with state-of-the-art methods, including SSCNet song2017semantic, 3D-Sketch chen20203d, and CleanerS wang2023semantic.
• Figure 5: Sensitivity analysis of key hyperparameters based on the random split validation set from NYU silberman2012indoor.