PIRLNav: Pretraining with Imitation and RL Finetuning for ObjectNav

TL;DR

PIRLNav combines imitation learning (behavior cloning on a large corpus of human demonstrations) with reinforcement learning finetuning to tackle ObjectNav. The two-stage approach uses a critic-first RL phase to stabilize training, followed by joint actor-critic optimization, achieving state-of-the-art results on HM3D ObjectNav. Through extensive ablations, the authors show human demonstrations yield superior RL transfer over automatically generated trajectories, and that while increasing BC data improves pretraining, RL gains eventually plateau, suggesting efficient data-size trade-offs. Failure mode analysis highlights dataset noise, inter-floor navigation, and recognition as key bottlenecks, pointing to data quality and navigation enhancements as priorities for future work.

Abstract

We study ObjectGoal Navigation -- where a virtual robot situated in a new environment is asked to navigate to an object. Prior work has shown that imitation learning (IL) using behavior cloning (BC) on a dataset of human demonstrations achieves promising results. However, this has limitations -- 1) BC policies generalize poorly to new states, since the training mimics actions not their consequences, and 2) collecting demonstrations is expensive. On the other hand, reinforcement learning (RL) is trivially scalable, but requires careful reward engineering to achieve desirable behavior. We present PIRLNav, a two-stage learning scheme for BC pretraining on human demonstrations followed by RL-finetuning. This leads to a policy that achieves a success rate of $65.0\%$ on ObjectNav ($+5.0\%$ absolute over previous state-of-the-art). Using this BC$\rightarrow$RL training recipe, we present a rigorous empirical analysis of design choices. First, we investigate whether human demonstrations can be replaced with `free' (automatically generated) sources of demonstrations, e.g. shortest paths (SP) or task-agnostic frontier exploration (FE) trajectories. We find that BC$\rightarrow$RL on human demonstrations outperforms BC$\rightarrow$RL on SP and FE trajectories, even when controlled for same BC-pretraining success on train, and even on a subset of val episodes where BC-pretraining success favors the SP or FE policies. Next, we study how RL-finetuning performance scales with the size of the BC pretraining dataset. We find that as we increase the size of BC-pretraining dataset and get to high BC accuracies, improvements from RL-finetuning are smaller, and that $90\%$ of the performance of our best BC$\rightarrow$RL policy can be achieved with less than half the number of BC demonstrations. Finally, we analyze failure modes of our ObjectNav policies, and present guidelines for further improving them.

Figures (8)

• Figure 1: ObjectNav success rates of agents trained using behavior cloning (BC) vs. BC-pretraining followed by reinforcement learning (RL) ( in blue). RL from scratch (i.e. BC=$0$) fails to get off-the-ground. With more BC demonstrations, BC success increases, and it transfers to even higher RL-finetuning success. But the difference between RL-finetuning vs. BC-pretraining success ( in orange) plateaus and starts to decrease beyond a certain point, indicating diminishing returns with each additional BC demonstration.
• Figure 2: ObjectNav trajectories for policies trained with BC$\rightarrow$RL on 1) Human Demonstrations, 2) Shortest Paths, and 3) Frontier Exploration Demonstrations.
• Figure 3: Learning rate schedule for RL Finetuning.
• Figure 4: ObjectNav performance on HM3D val with BC-pretraining on shortest path (SP), frontier exploration (FE), and human demonstrations (HD), followed by RL-finetuning from each.
• Figure 5: BC and RL performance for shortest paths (SP), frontier exploration (FE), and human demonstrations (HD) with equal BC training success on HM3Dtrain (left) and val (right).