APC-RL: Exceeding Data-Driven Behavior Priors with Adaptive Policy Composition

TL;DR

This work addresses the challenge of using demonstrations in reinforcement learning when those demonstrations are sparse, suboptimal, or misaligned. It introduces Adaptive Policy Composition (APC), a hierarchical architecture that combines multiple pre-trained Normalizing Flow priors with a prior-free actor under an adaptive, arbitrator-based selector, and a reward-sharing mechanism to ensure fair, data-efficient learning across actors. APC demonstrates robustness to misalignment, accelerates learning with aligned priors, and can exploit suboptimal demonstrations to bootstrap exploration, outperforming several baselines in diverse benchmarks. The approach offers a practical path to leveraging imperfect demonstrations in real-world tasks without sacrificing ultimately reward-driven adaptation, and it identifies key components—arbitrator selection and reward sharing—as critical for stable, efficient exploration.

Abstract

Incorporating demonstration data into reinforcement learning (RL) can greatly accelerate learning, but existing approaches often assume demonstrations are optimal and fully aligned with the target task. In practice, demonstrations are frequently sparse, suboptimal, or misaligned, which can degrade performance when these demonstrations are integrated into RL. We propose Adaptive Policy Composition (APC), a hierarchical model that adaptively composes multiple data-driven Normalizing Flow (NF) priors. Instead of enforcing strict adherence to the priors, APC estimates each prior's applicability to the target task while leveraging them for exploration. Moreover, APC either refines useful priors, or sidesteps misaligned ones when necessary to optimize downstream reward. Across diverse benchmarks, APC accelerates learning when demonstrations are aligned, remains robust under severe misalignment, and leverages suboptimal demonstrations to bootstrap exploration while avoiding performance degradation caused by overly strict adherence to suboptimal demonstrations.

Figures (16)

• Figure 1: Architecture overview. (a): PARROT levine2021parrot features a single latent policy and NF prior. (b): Our method uses a high-level selector to compose multiple latent policies and NF priors. A prior-free actor (index $(0)$, dashed border) learns directly in the action space. The selected latent policy and NF prior (cyan-colored arrows) are executed at time $t$. A reward-sharing trick (magenta-colored arrows) allows us to compute the latent coordinate $\mathbf{z}_t'$ corresponding to the executed action $\mathbf{a}_t$, and to use the transition at time $t$ to also update the other actors that were not selected.
• Figure 2: Our environmental testbed, from left to right: Maze Navigation, the different goals are marked in red. Franka Kitchen, with the manipulation targets marked in yellow. Car Racing.
• Figure 3: Time to success in the PointMaze Navigation environment. Each method was executed for at most 1.5M environment steps; each experiment was repeated with three random seeds. Bars indicate the step at which the cross-seed average running success rate reached 100%, or the final success rate after 1.5M steps if convergence was not achieved earlier (shorter bars are better). Percentage annotations denote the cross-seed average running success rate at that time (3 seeds).
• Figure 4: Results on FrankaKitchen's microwave task, which requires opening the microwave door. APC efficiently solves the task when exposed to aligned demonstrations (experiment (ii), top-left panel) while remaining robust under demonstration misalignment (experiment (i), remaining panels).
• Figure 5: Return curves on the car racing environment, the shaded area corresponds to one standard deviation around the mean, averaged over three seeds. Left: Experiment (iii) showing APC's performance relative to our baselines. Right: Experiment (iv) shows the performance of multiple APC ablations.