PEAfowl: Perception-Enhanced Multi-View Vision-Language-Action for Bimanual Manipulation

TL;DR

PEAfowl addresses robust bimanual manipulation under clutter and varying viewpoints by coupling geometry-guided multi-view fusion with a Perceiver-style language readout over frozen CLIP features. It introduces per-token depth distributions with differentiable 3D lifting and cross-view neighbor aggregation, plus depth distillation with a training-time depth teacher to inject geometry priors without test-time overhead. In RoboTwin 2.0 DR simulations and real-robot setups, PEAfowl achieves state-of-the-art success and demonstrates strong sim-to-real transfer, with ablations highlighting the importance of both geometry-guided perception and language-grounded readout. The approach advances robust, instruction-grounded manipulation in multi-view, cluttered environments, enabling more reliable cross-scene generalization and task grounding.

Abstract

Bimanual manipulation in cluttered scenes requires policies that remain stable under occlusions, viewpoint and scene variations. Existing vision-language-action models often fail to generalize because (i) multi-view features are fused via view-agnostic token concatenation, yielding weak 3D-consistent spatial understanding, and (ii) language is injected as global conditioning, resulting in coarse instruction grounding. In this paper, we introduce PEAfowl, a perception-enhanced multi-view VLA policy for bimanual manipulation. For spatial reasoning, PEAfowl predicts per-token depth distributions, performs differentiable 3D lifting, and aggregates local cross-view neighbors to form geometrically grounded, cross-view consistent representations. For instruction grounding, we propose to replace global conditioning with a Perceiver-style text-aware readout over frozen CLIP visual features, enabling iterative evidence accumulation. To overcome noisy and incomplete commodity depth without adding inference overhead, we apply training-only depth distillation from a pretrained depth teacher to supervise the depth-distribution head, providing perception front-end with geometry-aware priors. On RoboTwin 2.0 under domain-randomized setting, PEAfowl improves the strongest baseline by 23.0 pp in success rate, and real-robot experiments further demonstrate reliable sim-to-real transfer and consistent improvements from depth distillation. Project website: https://peafowlvla.github.io/.

Figures (7)

• Figure 1: Motivation and overview of PEAfowl. (a) Prior bimanual VLAs typically concatenate per-view visual tokens and apply global text conditioning, without explicit cross-view geometric alignment or instruction-relevant visual evidence retrieval and aggregation. (b) We propose PEAfowl, which incorporates geometry-guided multi-view fusion and a Perceiver-style text-as-query readout over frozen CLIP features. Bottom: Average success rates on RoboTwin 2.0 (nine training tasks) under Clean and Domain-Randomized settings, comparing multi-task bimanual baselines.
• Figure 2: PEAfowl architecture. PEAfowl couples geometry-guided multi-view fusion with language-guided readout to condition a SEM-style diffusion action decoder. Top: RGB–D tokens are used to predict per-token depth distributions for differentiable 3D lifting and cross-view fusion; a pretrained camera depth model supervises the depth-distribution head during training only. Bottom: Frozen CLIP features are queried by a Perceiver-style text-as-query readout and pooled into compact context tokens.
• Figure 3: Geometry-Guided Multi-View Fusion (GGMVF). Multi-scale RGB and depth features are tokenized, and co-located RGB–D pairs are used to predict discrete depth distributions for differentiable 3D lifting. The resulting 3D anchors enable top-$K$ cross-view neighbor aggregation in the base frame using distance-based softmax weights and a gated residual update. Aggregated RGB tokens are fused with depth tokens and 3D point embeddings via an MLP to produce geometry-enhanced tokens.
• Figure 4: Simulation and real-world setups. (a) RoboTwin 2.0 simulation (Aloha-AgileX, 4-camera RGB-D) under Clean (top) and Domain-Randomized (bottom) settings. (b) Dual-arm AgileX Piper with a 4-camera rig.
• Figure 5: Cross-view token consistency. Compared with SEM and PEAfowl pre-aggregation, PEAfowl post-aggregation yields more view-consistent, 3D-aligned clusters for the same physical regions across cameras.