Uncertainty-aware transfer across tasks using hybrid model-based successor feature reinforcement learning

TL;DR

This work addresses sample inefficiency in reinforcement learning by enabling transfer across tasks with different rewards and transition dynamics. It introduces UaMB-SF, a hybrid model-based and successor-feature framework that uses Kalman-filter–based multiple-model adaptive estimation to learn rewards and transition dynamics with uncertainty, enabling efficient SF-based value computation. The approach couples MB learning with SF representations to generalize knowledge across tasks while maintaining computational efficiency at decision time, and it incorporates uncertainty-aware exploration to further boost sample efficiency. Theoretical bounds on Q-value error and extensive experiments on continuous navigation and a combination-lock task demonstrate improved transfer performance and faster adaptation compared to strong baselines. The results suggest UaMB-SF as a practical framework for scalable, uncertainty-aware transfer learning in large or continuous state spaces.

Abstract

Sample efficiency is central to developing practical reinforcement learning (RL) for complex and large-scale decision-making problems. The ability to transfer and generalize knowledge gained from previous experiences to downstream tasks can significantly improve sample efficiency. Recent research indicates that successor feature (SF) RL algorithms enable knowledge generalization between tasks with different rewards but identical transition dynamics. It has recently been hypothesized that combining model-based (MB) methods with SF algorithms can alleviate the limitation of fixed transition dynamics. Furthermore, uncertainty-aware exploration is widely recognized as another appealing approach for improving sample efficiency. Putting together two ideas of hybrid model-based successor feature (MB-SF) and uncertainty leads to an approach to the problem of sample efficient uncertainty-aware knowledge transfer across tasks with different transition dynamics or/and reward functions. In this paper, the uncertainty of the value of each action is approximated by a Kalman filter (KF)-based multiple-model adaptive estimation. This KF-based framework treats the parameters of a model as random variables. To the best of our knowledge, this is the first attempt at formulating a hybrid MB-SF algorithm capable of generalizing knowledge across large or continuous state space tasks with various transition dynamics while requiring less computation at decision time than MB methods. The number of samples required to learn the tasks was compared to recent SF and MB baselines. The results show that our algorithm generalizes its knowledge across different transition dynamics, learns downstream tasks with significantly fewer samples than starting from scratch, and outperforms existing approaches.

Figures (16)

• Figure 1: An example for comparing the behaviours of TD-SR and MB-SR after changes in the transition dynamics $P$.
• Figure 2: The structure of the multiple-model adaptive estimation with $M_{\text{KF}}$ parallel KFs used for reward function learning.
• Figure 3: Visual Illustration of the proposed $\text{UaMB-SF}$ framework.
• Figure 4: Transfer learning of the proposed $\text{UaMB-SF}$ framework: once the agent learns the source task, the trained parameters are kept and transferred to initialize the parameters of $\text{UaMB-SF}$ for learning the test task, which differs from the source task in the reward or transition dynamics at $\left<\bm{s}',a', R(\bm{s}',a'),\bm{s}"\right>$. During the test task learning process, the agent mimics the actions learned in the source task until it encounters the change.
• Figure 5: Goal-oriented source tasks A and 1.