A General One-Shot Multimodal Active Perception Framework for Robotic Manipulation: Learning to Predict Optimal Viewpoint

TL;DR

This work tackles the challenge of selecting informative viewpoints for vision-based robotic manipulation by proposing a general one-shot multimodal active perception framework. The core idea is to decouple viewpoint quality evaluation from the architecture and to learn a multimodal viewpoint predictor (MVPNet) that directly outputs camera pose adjustments $ (Delta t, Delta q) $ via cross-attention over mask and point-cloud features. A large synthetic dataset is built with domain randomization to label optimal viewpoints, enabling robust one-shot prediction and transfer to real-world settings. Empirical results in simulation show consistent improvement across various grasp-pose estimators, and real-world experiments demonstrate near-doubling of grasp success with seamless sim-to-real transfer without extra fine-tuning. The framework thus offers generality and practical impact for rapid, transfer-ready active perception in robotic manipulation and beyond.

Abstract

Active perception in vision-based robotic manipulation aims to move the camera toward more informative observation viewpoints, thereby providing high-quality perceptual inputs for downstream tasks. Most existing active perception methods rely on iterative optimization, leading to high time and motion costs, and are tightly coupled with task-specific objectives, which limits their transferability. In this paper, we propose a general one-shot multimodal active perception framework for robotic manipulation. The framework enables direct inference of optimal viewpoints and comprises a data collection pipeline and an optimal viewpoint prediction network. Specifically, the framework decouples viewpoint quality evaluation from the overall architecture, supporting heterogeneous task requirements. Optimal viewpoints are defined through systematic sampling and evaluation of candidate viewpoints, after which large-scale training datasets are constructed via domain randomization. Moreover, a multimodal optimal viewpoint prediction network is developed, leveraging cross-attention to align and fuse multimodal features and directly predict camera pose adjustments. The proposed framework is instantiated in robotic grasping under viewpoint-constrained environments. Experimental results demonstrate that active perception guided by the framework significantly improves grasp success rates. Notably, real-world evaluations achieve nearly double the grasp success rate and enable seamless sim-to-real transfer without additional fine-tuning, demonstrating the effectiveness of the proposed framework.

Figures (7)

• Figure 1: Conceptual illustration of the proposed "Focus-then-Execute" active perception paradigm.
• Figure 2: Overall framework of the proposed method, illustrated with robotic grasping in viewpoint-constrained environments: (a) sampling and evaluating candidate viewpoints to obtain the optimal viewpoint for each object, followed by dataset construction via domain randomization; (b) training the MVPNet based on the constructed dataset; and (c) deploying the trained network and conducting comparative evaluations.
• Figure 3: (a) Simulation setup; (b) Similar objects; (c) Novel objects. The simulation environment based on Isaac Sim serves for synthetic dataset construction and simulation-based experimental testing.
• Figure 4: An example of the viewpoint score distribution of the object: (a) 3D distribution; (b) X-Y plane projection; (c) X-Z plane projection; (d) Y-Z plane projection. Each point represents a observation viewpoint, with color indicating its score: red (highest), green (medium), and blue (lowest).
• Figure 5: Overall framework of the perception and preprocessing modules together with MVPNet.

Theorems & Definitions (1)

• Remark 1