Improving Viewpoint-Independent Object-Centric Representations through Active Viewpoint Selection

TL;DR

The effectiveness of this active viewpoint selection strategy is demonstrated, significantly enhancing segmentation and reconstruction performance compared to random viewpoint selection and it can accurately predict images from unknown viewpoints.

Abstract

Given the complexities inherent in visual scenes, such as object occlusion, a comprehensive understanding often requires observation from multiple viewpoints. Existing multi-viewpoint object-centric learning methods typically employ random or sequential viewpoint selection strategies. While applicable across various scenes, these strategies may not always be ideal, as certain scenes could benefit more from specific viewpoints. To address this limitation, we propose a novel active viewpoint selection strategy. This strategy predicts images from unknown viewpoints based on information from observation images for each scene. It then compares the object-centric representations extracted from both viewpoints and selects the unknown viewpoint with the largest disparity, indicating the greatest gain in information, as the next observation viewpoint. Through experiments on various datasets, we demonstrate the effectiveness of our active viewpoint selection strategy, significantly enhancing segmentation and reconstruction performance compared to random viewpoint selection. Moreover, our method can accurately predict images from unknown viewpoints.

Figures (13)

• Figure 1: Active viewpoint selection framework. Our proposed method iteratively selects viewpoints from the unknown set to form a small yet informative observation set, enabling effective training with fewer images. The active viewpoint selection strategy evaluates the information gain of the unknown viewpoints using the predicted images and selects the viewpoint with the maximum information gain as the next observation. The real image of the selected viewpoint is then added to the observation set, and this process continues until the observation set reaches a predefined size.
• Figure 2: Model architecture overview. Given the observation set $\mathcal{O}$, model learns viewpoint-independent object-centric representations $S^{\text{prev}}$ from Multi-Viewpoint Slot Attention and viewpoint representations $S^{view}_\mathcal{O}$ from Viewpoint Encoder. These representations are concatenated and input into the Diffusion-base Decoder to reconstruct the observation set. For the unknown set $\mathcal{P}$, model obtains viewpoint representations $S^{view}_{\mathcal{P}}$ from Viewpoint Encoder. $S^{\text{prev}}$ and $S^{view}_{\mathcal{P}}$ are concatenated and input into the Diffusion-base Decoder to predict images. The object representations $S^{\text{new}}$ are obtained from the predicted image and compared with $S^{\text{prev}}$ to evaluate the information gain of the unknown viewpoint. The viewpoint with the maximum information gain is selected and its corresponding real image $\boldsymbol{x}_{sel}$ is added to the observation set.
• Figure 3: Visualization of segmentation results on CLEVRTEX and GSO.
• Figure 4: Visualization of reconstruction results on CLEVRTEX, GSO, and ShapeNet.
• Figure 5: Multi-viewpoint compositional generation samples and interpolation.