Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Robust Point Cloud Reinforcement Learning via PCA-Based Canonicalization

Michael Bezick, Vittorio Giammarino, Ahmed H. Qureshi

TL;DR

The proposed PCA Point Cloud (PPC) improves robustness to unseen camera poses across challenging robotic tasks, providing a principled alternative to domain randomization.

Abstract

Reinforcement Learning (RL) from raw visual input has achieved impressive successes in recent years, yet it remains fragile to out-of-distribution variations such as changes in lighting, color, and viewpoint. Point Cloud Reinforcement Learning (PC-RL) offers a promising alternative by mitigating appearance-based brittleness, but its sensitivity to camera pose mismatches continues to undermine reliability in realistic settings. To address this challenge, we propose PCA Point Cloud (PPC), a canonicalization framework specifically tailored for downstream robotic control. PPC maps point clouds under arbitrary rigid-body transformations to a unique canonical pose, aligning observations to a consistent frame, thereby substantially decreasing viewpoint-induced inconsistencies. In our experiments, we show that PPC improves robustness to unseen camera poses across challenging robotic tasks, providing a principled alternative to domain randomization.

Robust Point Cloud Reinforcement Learning via PCA-Based Canonicalization

TL;DR

The proposed PCA Point Cloud (PPC) improves robustness to unseen camera poses across challenging robotic tasks, providing a principled alternative to domain randomization.

Abstract

Reinforcement Learning (RL) from raw visual input has achieved impressive successes in recent years, yet it remains fragile to out-of-distribution variations such as changes in lighting, color, and viewpoint. Point Cloud Reinforcement Learning (PC-RL) offers a promising alternative by mitigating appearance-based brittleness, but its sensitivity to camera pose mismatches continues to undermine reliability in realistic settings. To address this challenge, we propose PCA Point Cloud (PPC), a canonicalization framework specifically tailored for downstream robotic control. PPC maps point clouds under arbitrary rigid-body transformations to a unique canonical pose, aligning observations to a consistent frame, thereby substantially decreasing viewpoint-induced inconsistencies. In our experiments, we show that PPC improves robustness to unseen camera poses across challenging robotic tasks, providing a principled alternative to domain randomization.
Paper Structure (19 sections, 3 theorems, 12 equations, 4 figures, 1 table)

This paper contains 19 sections, 3 theorems, 12 equations, 4 figures, 1 table.

Table of Contents

  1. Introduction
  2. Related Work
  3. Point Cloud Representation Learning
  4. Point Cloud Reinforcement Learning
  5. Point Cloud Rotation and Translation Invariance
  6. Preliminaries
  7. Partially Observable Markov Decision Process
  8. Reinforcement Learning
  9. Point Patch Reinforcement Learning
  10. Method
  11. Point Cloud Downsampling
  12. PCA Point Cloud
  13. Overview
  14. Theoretical Results
  15. PPC in RL
  16. ...and 4 more sections

Key Result

Lemma 1

Let $\mathcal{E}(x) = \{\bm E(x)\,\mathrm{diag}(\pm1,\pm1,\pm1)\}$ and $\mathcal{E}(x') = \{\bm E(x')\,\mathrm{diag}(\pm1,\pm1,\pm1)\}$ denote the sets of orthogonal eigenvector bases obtained from point sets $x$ and $x'$ respectively. Assume a simple spectrum $\lambda_1>\lambda_2>\lambda_3\ge 0$. I

Figures (4)

  • Figure 1: Illustration of the proposed PCA Point Cloud (PPC) pipeline. Given an input point cloud, PPC first applies voxel downsampling to regularize point distribution, followed by Farthest Point Sampling (FPS) to select a fixed number of representative points, ensuring consistent shape coverage and enabling uniform batch processing. A PCA-based canonicalization is then performed, producing four candidate eigenvector bases, from which a geometry-driven disambiguation step deterministically selects one. The point cloud is subsequently re-expressed in this basis, yielding a unique disambiguated canonical pose. As a result, regardless of the initial rotation or translation of the input, the final canonical representation remains consistent.
  • Figure 2: Environments for zero-shot generalization in PC-RL.
  • Figure 3: Summary of zero-shot generalization experiments in PC-RL. For each environment, the first plot depicts the algorithms' performance during training, while the others evaluate algorithms for zero-shot generalization under camera viewpoint mismatch. We run ThreadInHole for $5 \times10^5$ steps, DeflectSpheres for $10^6$ steps, and TurnFaucet for $7.5 \times 10^5$ steps, all for $6$ seeds. We evaluate over five episodes every $100$k steps, and plot means with 95% bootstrapped confidence intervals across episodes and seeds. We use the auxiliary loss in PPRL for all tasks except DeflectSpheres, following the observation by Gyenes et al. gyenes2024pointpatchrl that it reduces performance in this setting.
  • Figure 4: We visualize the selected test set of mismatches for the experiments in Table \ref{['tab:real_world_test']}, showing the original view in orange and the perturbed view in RGB.

Theorems & Definitions (7)

  • Lemma 1: Translation invariance and rotation equivariance
  • proof
  • Lemma 2: Sign consistency
  • proof
  • Theorem 1: Rigid-motion invariance on the non-degenerate simple-spectrum set
  • proof
  • Remark 1: Degeneracies