Tiered Reward: Designing Rewards for Specification and Fast Learning of Desired Behavior

TL;DR

The paper tackles reward-design in reinforcement learning, focusing on expressing desirable states and avoiding undesirable ones in goal-obstacle tasks. It introduces Tiered Reward, a class of environment-independent rewards defined over $k$ tiers, which induces Pareto-optimal policies under a strict partial order on outcomes. The authors prove sufficient conditions for 3-tier and general $k$-tier rewards to guarantee Pareto-optimality and demonstrate faster learning in both tabular and deep RL across multiple domains, including grid worlds and MiniGrid. Empirical results show Tiered Reward consistently accelerates learning relative to baselines like action penalty and tier-based shaping, with improvements that generalize across algorithms such as Q-learning and PPO, and reveal nuances in tier count due to scaling. The work reduces environment-specific reward engineering and offers a practical, algorithm-agnostic approach to guiding fast, correct behavior in RL tasks.

Abstract

Reinforcement-learning agents seek to maximize a reward signal through environmental interactions. As humans, our job in the learning process is to design reward functions to express desired behavior and enable the agent to learn such behavior swiftly. However, designing good reward functions to induce the desired behavior is generally hard, let alone the question of which rewards make learning fast. In this work, we introduce a family of a reward structures we call Tiered Reward that addresses both of these questions. We consider the reward-design problem in tasks formulated as reaching desirable states and avoiding undesirable states. To start, we propose a strict partial ordering of the policy space to resolve trade-offs in behavior preference. We prefer policies that reach the good states faster and with higher probability while avoiding the bad states longer. Next, we introduce Tiered Reward, a class of environment-independent reward functions and show it is guaranteed to induce policies that are Pareto-optimal according to our preference relation. Finally, we demonstrate that Tiered Reward leads to fast learning with multiple tabular and deep reinforcement-learning algorithms.

Figures (11)

• Figure 1:
• Figure 2: Tiered Reward leads to fast learning on the Flag Grid.
• Figure 3: Illustration of EmptyGrid, FourRooms, and DoorKey from left to right. The agent is in red, and the goal in green. The objective is to navigate to the goal; in FourRooms, the agent has to find the gaps in the wall; in DoorKey, the agent must first pick up the key and unlock the yellow door. All three environments use visual observations.
• Figure 4: Learning curves of three reward functions on EmptyGrid, FourRooms, and DoorKey. Each agent is trained with the reward function labeled on the plot, but evaluated using the original MiniGrid reward ($1 - 0.9 * \text{step count} / \text{max steps}$ for success, and $0$ for failure). Error bars show standard deviation from $30$ random seeds.
• Figure 5: Influence of more tiers on four grid worlds with Q learning. RMAX results in Figure \ref{['fig:chain-grid-rmax']}.

Theorems & Definitions (8)

• Definition 3.1
• Definition 5.1
• Definition 5.2
• Theorem 5.3: Pareto-optimal rewards in 3-Tier MDP
• Definition 5.4
• Theorem 5.5: Tiered Reward and Cumulative Tier Visitation
• proof
• proof