Posterior Behavioral Cloning: Pretraining BC Policies for Efficient RL Finetuning

TL;DR

This work shows that standard BC can underfit the demonstrator's action distribution, limiting RL finetuning. It introduces Posterior Behavioral Cloning (PostBc), which models the posterior over the demonstrator's behavior and mixes it with BC to achieve demonstrator action coverage without degrading pretrained performance. Theoretical results establish near-optimal coverage guarantees and suboptimality bounds, while practical instantiations using ensembles and diffusion policies demonstrate improved RL finetuning efficiency on simulated and real robotic tasks, including multi-task Libero and WidowX. Overall, PostBc offers a scalable, model-agnostic pretraining strategy that enhances downstream learning while retaining or improving the initial pretrained policy performance.

Abstract

Standard practice across domains from robotics to language is to first pretrain a policy on a large-scale demonstration dataset, and then finetune this policy, typically with reinforcement learning (RL), in order to improve performance on deployment domains. This finetuning step has proved critical in achieving human or super-human performance, yet while much attention has been given to developing more effective finetuning algorithms, little attention has been given to ensuring the pretrained policy is an effective initialization for RL finetuning. In this work we seek to understand how the pretrained policy affects finetuning performance, and how to pretrain policies in order to ensure they are effective initializations for finetuning. We first show theoretically that standard behavioral cloning (BC) -- which trains a policy to directly match the actions played by the demonstrator -- can fail to ensure coverage over the demonstrator's actions, a minimal condition necessary for effective RL finetuning. We then show that if, instead of exactly fitting the observed demonstrations, we train a policy to model the posterior distribution of the demonstrator's behavior given the demonstration dataset, we do obtain a policy that ensures coverage over the demonstrator's actions, enabling more effective finetuning. Furthermore, this policy -- which we refer to as the posterior behavioral cloning (PostBC) policy -- achieves this while ensuring pretrained performance is no worse than that of the BC policy. We then show that PostBC is practically implementable with modern generative models in robotic control domains -- relying only on standard supervised learning -- and leads to significantly improved RL finetuning performance on both realistic robotic control benchmarks and real-world robotic manipulation tasks, as compared to standard behavioral cloning.

Figures (13)

• Figure 1: (a) We consider the setting where we are given demonstration data for some tasks of interest. (b) Standard Bc pretraining fits the behaviors in the demonstrations, leading to effective performance in regions with high demonstration data density, yet can overcommit to the observed behaviors in regions with low data density. (c) This leads to ineffective RL finetuning, since rollouts from the Bc policy provide little meaningful reward signal in such low data density regions, which is typically necessary to enable effective improvement. (d) In contrast, we propose posterior behavioral cloning (PostBc), which instead of directly mimicking the demonstrations, trains a generative policy to fit the posterior distribution of the demonstrator's behavior. This endows the pretrained policy with a wider distribution of actions in regions of low demonstrator data density, while in regions of high data density it reduces to approximately the standard Bc policy. (e) This wider action distribution in low data density regions allows for collection of diverse observations with more informative reward signal, enabling more effective RL finetuning, while in regions of high data density performance converges to that of the demonstrator.
• Figure 2: Robomimic and Libero settings
• Figure 3: Comparison of Dsrl finetuning performance combined with different Bc pretraining approaches on Robomimic.
• Figure 4: Comparison of Dppo finetuning performance combined with different Bc pretraining approaches on Robomimic.
• Figure 5: Comparison of Dsrl finetuning performance combined with different Bc pretraining approaches on all tasks from Libero 90, Kitchen Scene 2.

Theorems & Definitions (27)

• Proposition 1: rajaraman2020toward
• Definition 4.1: Demonstrator Action Coverage
• Proposition 2
• Proposition 3
• Definition 4.2: Posterior Demonstrator Policy
• Theorem 1
• Theorem 2
• Proposition 4
• proof : Proof of \ref{['prop:bc_fails']}
• proof : Proof of \ref{['prop:unif_fails']}