NVSPolicy: Adaptive Novel-View Synthesis for Generalizable Language-Conditioned Policy Learning

TL;DR

The paper tackles long-horizon, language-conditioned robotic manipulation under partial observations by introducing NVSPolicy, which fuses adaptive novel-view synthesis with a cycle-consistent latent disentanglement and a hierarchical policy. It advances the field with an adaptive viewpoint selector that uses a local spherical coordinate system to compute $\theta = -w_1 \cdot d^{\text{cam-ori}} + w_2$ (with $w_1=14$, $w_2=39$) and leverages GenWarp to generate informative, viewpoint-consistent novel views $I^{\text{synthesis}}$, alongside a cycle-consistent VAE that separates semantic $s$ from remaining $r$ features. The hierarchical policy then uses $s$ to select a high-level meta-skill and $r$ to predict low-level actions, with practical efficiency via keyframe synthesis and policy distillation that preserves semantic integrity with reduced computation. Empirically, NVSPolicy achieves state-of-the-art results on CALVIN with an average success rate of $90.4\%$ and a mean long-horizon completion capacity of $2.93$, and real-world experiments corroborate sim-to-real robustness, highlighting its practical impact for robust, open-world robotic manipulation.

Abstract

Recent advances in deep generative models demonstrate unprecedented zero-shot generalization capabilities, offering great potential for robot manipulation in unstructured environments. Given a partial observation of a scene, deep generative models could generate the unseen regions and therefore provide more context, which enhances the capability of robots to generalize across unseen environments. However, due to the visual artifacts in generated images and inefficient integration of multi-modal features in policy learning, this direction remains an open challenge. We introduce NVSPolicy, a generalizable language-conditioned policy learning method that couples an adaptive novel-view synthesis module with a hierarchical policy network. Given an input image, NVSPolicy dynamically selects an informative viewpoint and synthesizes an adaptive novel-view image to enrich the visual context. To mitigate the impact of the imperfect synthesized images, we adopt a cycle-consistent VAE mechanism that disentangles the visual features into the semantic feature and the remaining feature. The two features are then fed into the hierarchical policy network respectively: the semantic feature informs the high-level meta-skill selection, and the remaining feature guides low-level action estimation. Moreover, we propose several practical mechanisms to make the proposed method efficient. Extensive experiments on CALVIN demonstrate the state-of-the-art performance of our method. Specifically, it achieves an average success rate of 90.4\% across all tasks, greatly outperforming the recent methods. Ablation studies confirm the significance of our adaptive novel-view synthesis paradigm. In addition, we evaluate NVSPolicy on a real-world robotic platform to demonstrate its practical applicability.

Figures (6)

• Figure 1: The proposed NVSPolicy leverages adaptive novel-view synthesis to provide more context, boosting the performance of language-conditioned policy learning.
• Figure 2: The overview of the proposed NVSPolicy. (a) It synthesizes context-enhanced novel-view images given the input images. (b) It disentangles the image into the semantic feature and the remaining feature with a cycle-consistent VAE mechanism, mitigating the impact of the imperfect synthesized images. (c) Based on (a) and (b), a hierarchical policy network is developed to estimate the optimal meta-skill and the low-level action.
• Figure 3: (a) The local spherical coordinate in the adaptive novel viewpoint selection. (b) Examples of the synthesized image.
• Figure 4: The training protocol of the cycle-consistent VAE: a forward process and a reverse process are developed to jointly optimize the encoder and decoder. Once trained, the encoder can be applied to disentangle the visual features into the semantic feature and the remaining feature during inference.
• Figure 5: Qualitative comparisons of NVSPolicy with several recent methods. Frames marked with a check mark indicate a successful task completion, while the representative failure cases are highlighted with red dashed boxes.