Multi-CALF: A Policy Combination Approach with Statistical Guarantees
Georgiy Malaniya, Anton Bolychev, Grigory Yaremenko, Anastasia Krasnaya, Pavel Osinenko
TL;DR
Multi-CALF tackles the tension between reward-driven performance and formal stability in reinforcement learning by fusing a base policy that optimizes rewards with an alternative policy that guarantees goal-reaching. A critic-based switching mechanism uses relative value improvements and a decaying acceptance probability to ensure the alternative policy eventually governs control, yielding convergence to the goal set $\mathbb{G}$ with probability at least $1-\varepsilon$ and providing bounds on convergence time and deviation. The authors establish theoretical guarantees under reasonable smoothness and boundedness assumptions and demonstrate empirical gains on control tasks (e.g., Hopper) where the fused policy outperforms both components. The approach is lightweight, compatible with standard RL libraries, and offers a practical path to safe, high-performance deployment; extending to more policies and adaptive policy discovery are promising future directions.
Abstract
We introduce Multi-CALF, an algorithm that intelligently combines reinforcement learning policies based on their relative value improvements. Our approach integrates a standard RL policy with a theoretically-backed alternative policy, inheriting formal stability guarantees while often achieving better performance than either policy individually. We prove that our combined policy converges to a specified goal set with known probability and provide precise bounds on maximum deviation and convergence time. Empirical validation on control tasks demonstrates enhanced performance while maintaining stability guarantees.