A Unified Framework for Zero-Shot Reinforcement Learning

TL;DR

This paper addresses the fragmentation in zero-shot reinforcement learning by proposing a unified framework that categorizes methods along two axes: direct versus compositional representations and reward-free versus pseudo reward-free training. It formalizes the zero-shot objective through an MDP family over reward functions and highlights successor-based approaches as a central, transferable structure, deriving an extended bound for successor features in the zero-shot regime. The work clarifies how direct methods learn end-to-end mappings from rewards to values, while compositional methods exploit value-function structure (e.g., successor measures, forward-backward representations) to enable test-time generalization and faster inference. By articulating a principled taxonomy, the framework guides future research toward more general agents and unifies disparate strands of zero-shot RL literature, including reward embeddings, Hilbert representations, and proto successor measures. The practical impact lies in providing a coherent blueprint for designing and evaluating zero-shot agents that can adapt to a broad set of downstream objectives without additional training or planning at test time.

Abstract

Zero-shot reinforcement learning (RL) has emerged as a setting for developing general agents in an unsupervised manner, capable of solving downstream tasks without additional training or planning at test-time. Unlike conventional RL, which optimizes policies for a fixed reward, zero-shot RL requires agents to encode representations rich enough to support immediate adaptation to any objective, drawing parallels to vision and language foundation models. Despite growing interest, the field lacks a common analytical lens. We present the first unified framework for zero-shot RL. Our formulation introduces a consistent notation and taxonomy that organizes existing approaches and allows direct comparison between them. Central to our framework is the classification of algorithms into two families: direct representations, which learn end-to-end mappings from rewards to policies, and compositional representations, which decompose the representation leveraging the substructure of the value function. Within this framework, we highlight shared principles and key differences across methods, and we derive an extended bound for successor-feature methods, offering a new perspective on their performance in the zero-shot regime. By consolidating existing work under a common lens, our framework provides a principled foundation for future research in zero-shot RL and outlines a clear path toward developing more general agents.

Figures (2)

• Figure 1: Our proposed taxonomy for zero-shot reinforcement learning distinguishes between direct and compositional methods. Direct methods learn action–value functions end-to-end, while compositional methods exploit structural decompositions, learning intermediate quantities that can later be used to infer values. We further differentiate between approaches with reward-independent training objectives—relying solely on transition dynamics and policy—and those that generalize across reward functions by training on collections of (possibly random) rewards to enable adaptation at inference time.
• Figure 2: Comparison between pseudo reward-free and reward-free methods in terms of test-time computation. Pseudo–reward-free methods offload most of the computational cost to the pretraining phase, enabling efficient inference. In contrast, existing reward-free approaches rely on explicit search during inference, resulting in higher test-time computation. The dotted zero-shot boundary indicates that there is no fixed threshold defining zero-shot computation. Under current formulations, only pseudo–reward-free methods can be unambiguously classified as zero-shot, as they require no explicit search in the space of solutions.

Theorems & Definitions (4)

• lemma 1: barreto_successor_2017
• theorem 1: Generalized Policy Improvement barreto_successor_2017
• theorem 2
• theorem 3: Theoren \ref{['thm:2']}