Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Viewpoint Matters: Dynamically Optimizing Viewpoints with Masked Autoencoder for Visual Manipulation

Pengfei Yi, Yifan Han, Junyan Li, Litao Liu, Wenzhao Lian

TL;DR

This paper addresses the rigidity of fixed camera setups in imitation-learning-based robotic manipulation by introducing MAE-Select, a framework that uses pre-trained multi-view masked autoencoder representations to actively select the next best viewpoint at each time chunk for a single-camera system. The method jointly learns an action policy and a viewpoint selector in a label-free manner, leveraging the MV-MAE latent context and a diffusion-based action decoder, with a reconstruction loss guiding representation learning. Empirical results across simulated and real-world tasks show MAE-Select improves performance over fixed single-camera setups and can match or exceed multi-camera configurations in several tasks, while ablations highlight the importance of the full encoder–decoder MV-MAE. The work demonstrates the practical viability of dynamic viewpoint optimization in robotics, and points to future directions in continuous-viewpoint optimization using NeRF or 3D Gaussian processes.

Abstract

Robotic manipulation continues to be a challenge, and imitation learning (IL) enables robots to learn tasks from expert demonstrations. Current IL methods typically rely on fixed camera setups, where cameras are manually positioned in static locations, imposing significant limitations on adaptability and coverage. Inspired by human active perception, where humans dynamically adjust their viewpoint to capture the most relevant and least noisy information, we propose MAE-Select, a novel framework for active viewpoint selection in single-camera robotic systems. MAE-Select fully leverages pre-trained multi-view masked autoencoder representations and dynamically selects the next most informative viewpoint at each time chunk without requiring labeled viewpoints. Extensive experiments demonstrate that MAE-Select improves the capabilities of single-camera systems and, in some cases, even surpasses multi-camera setups. The project will be available at https://mae-select.github.io.

Viewpoint Matters: Dynamically Optimizing Viewpoints with Masked Autoencoder for Visual Manipulation

TL;DR

This paper addresses the rigidity of fixed camera setups in imitation-learning-based robotic manipulation by introducing MAE-Select, a framework that uses pre-trained multi-view masked autoencoder representations to actively select the next best viewpoint at each time chunk for a single-camera system. The method jointly learns an action policy and a viewpoint selector in a label-free manner, leveraging the MV-MAE latent context and a diffusion-based action decoder, with a reconstruction loss guiding representation learning. Empirical results across simulated and real-world tasks show MAE-Select improves performance over fixed single-camera setups and can match or exceed multi-camera configurations in several tasks, while ablations highlight the importance of the full encoder–decoder MV-MAE. The work demonstrates the practical viability of dynamic viewpoint optimization in robotics, and points to future directions in continuous-viewpoint optimization using NeRF or 3D Gaussian processes.

Abstract

Robotic manipulation continues to be a challenge, and imitation learning (IL) enables robots to learn tasks from expert demonstrations. Current IL methods typically rely on fixed camera setups, where cameras are manually positioned in static locations, imposing significant limitations on adaptability and coverage. Inspired by human active perception, where humans dynamically adjust their viewpoint to capture the most relevant and least noisy information, we propose MAE-Select, a novel framework for active viewpoint selection in single-camera robotic systems. MAE-Select fully leverages pre-trained multi-view masked autoencoder representations and dynamically selects the next most informative viewpoint at each time chunk without requiring labeled viewpoints. Extensive experiments demonstrate that MAE-Select improves the capabilities of single-camera systems and, in some cases, even surpasses multi-camera setups. The project will be available at https://mae-select.github.io.
Paper Structure (10 sections, 5 equations, 2 figures, 3 tables)

This paper contains 10 sections, 5 equations, 2 figures, 3 tables.

Table of Contents

  1. Introduction
  2. Method
  3. Problem Formulation
  4. Multi-View Masked Autoencoder Pre-training
  5. Next Better Viewpoint Selection
  6. Experiments
  7. Implementation
  8. Results
  9. Ablation Study
  10. Conclusions and Future Work

Figures (2)

  • Figure 1: Illustration of our proposed method. Left depicts the pre-training stage of the multi-view masked autoencoder with joint embeddings. Middle illustrates the training process of our framework using imitation learning. Right demonstrates how the framework operates during inference.
  • Figure 2: Visualization of the selected viewpoints in our experiments, showcasing both simulation and real-world environments. Each row represents the procedure of a specific task, indicating the necessity of selecting different viewpoints.