Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Informativeness of Reward Functions in Reinforcement Learning

Rati Devidze, Parameswaran Kamalaruban, Adish Singla

TL;DR

This work tackles how to design reward functions in reinforcement learning that are informative and interpretable, with a focus on expert-driven settings. It introduces ExpAdaRD, a framework that optimizes an informativeness criterion I_L(R) within a bi-level formulation while enforcing invariance and structural constraints to preserve task semantics and interpretability; it also proposes a policy-agnostic version I_h to enable algorithm-agnostic assessment. The authors show theoretically that maximizing I_h can accelerate convergence toward a target policy and provide a convergence guarantee in a simplified setting. Empirically, ExpAdaRD improves learning speed and yields adaptive, interpretable rewards in two navigation tasks (Room and LineK) compared with nonadaptive baselines and prior explicable reward designs. Overall, the approach demonstrates the practical value of policy-aware reward informativity for faster, more transparent RL training in structured tasks, with potential extensions to richer environments and automated constraint discovery.

Abstract

Reward functions are central in specifying the task we want a reinforcement learning agent to perform. Given a task and desired optimal behavior, we study the problem of designing informative reward functions so that the designed rewards speed up the agent's convergence. In particular, we consider expert-driven reward design settings where an expert or teacher seeks to provide informative and interpretable rewards to a learning agent. Existing works have considered several different reward design formulations; however, the key challenge is formulating a reward informativeness criterion that adapts w.r.t. the agent's current policy and can be optimized under specified structural constraints to obtain interpretable rewards. In this paper, we propose a novel reward informativeness criterion, a quantitative measure that captures how the agent's current policy will improve if it receives rewards from a specific reward function. We theoretically showcase the utility of the proposed informativeness criterion for adaptively designing rewards for an agent. Experimental results on two navigation tasks demonstrate the effectiveness of our adaptive reward informativeness criterion.

Informativeness of Reward Functions in Reinforcement Learning

TL;DR

This work tackles how to design reward functions in reinforcement learning that are informative and interpretable, with a focus on expert-driven settings. It introduces ExpAdaRD, a framework that optimizes an informativeness criterion I_L(R) within a bi-level formulation while enforcing invariance and structural constraints to preserve task semantics and interpretability; it also proposes a policy-agnostic version I_h to enable algorithm-agnostic assessment. The authors show theoretically that maximizing I_h can accelerate convergence toward a target policy and provide a convergence guarantee in a simplified setting. Empirically, ExpAdaRD improves learning speed and yields adaptive, interpretable rewards in two navigation tasks (Room and LineK) compared with nonadaptive baselines and prior explicable reward designs. Overall, the approach demonstrates the practical value of policy-aware reward informativity for faster, more transparent RL training in structured tasks, with potential extensions to richer environments and automated constraint discovery.

Abstract

Reward functions are central in specifying the task we want a reinforcement learning agent to perform. Given a task and desired optimal behavior, we study the problem of designing informative reward functions so that the designed rewards speed up the agent's convergence. In particular, we consider expert-driven reward design settings where an expert or teacher seeks to provide informative and interpretable rewards to a learning agent. Existing works have considered several different reward design formulations; however, the key challenge is formulating a reward informativeness criterion that adapts w.r.t. the agent's current policy and can be optimized under specified structural constraints to obtain interpretable rewards. In this paper, we propose a novel reward informativeness criterion, a quantitative measure that captures how the agent's current policy will improve if it receives rewards from a specific reward function. We theoretically showcase the utility of the proposed informativeness criterion for adaptively designing rewards for an agent. Experimental results on two navigation tasks demonstrate the effectiveness of our adaptive reward informativeness criterion.
Paper Structure (28 sections, 2 theorems, 9 equations, 4 figures, 2 algorithms)

This paper contains 28 sections, 2 theorems, 9 equations, 4 figures, 2 algorithms.

Table of Contents

  1. Introduction
  2. Related Work
  3. Expert-driven reward design.
  4. Learner-driven reward design.
  5. Preliminaries
  6. Environment.
  7. Policy and performance.
  8. Expert-driven Explicable and Adaptive Reward Design
  9. Invariance constraints.
  10. Structural constraints.
  11. Informativeness Criterion for Reward Design
  12. Notation.
  13. Bi-Level Formulation for Reward Informativeness $I_L(R)$
  14. Analysis for a specific learning algorithm $L$.
  15. Intuitive Formulation for Reward Informativeness $I_h(R)$
  16. ...and 13 more sections

Key Result

Proposition 1

The gradient of the informativeness criterion in Eq. eq:intuitive-IR-bi-level for the simplified learning algorithm $L$ with $h$-depth planning described above takes the following form: where ${A}^{\pi^T}_{\overline{R}}(s, \pi^L(s)) = \mathbb{E}_{\pi^L(a'|s)}\!\left[{A}^{\pi^T}_{\overline{R}}(s, a')\right]$, and ${A}^{\pi^L}_{R_\phi,h} (s,a) = {Q}^{\pi^L}_{R_\phi,h} (s,a) - {V}^{\pi^L}_{R_\phi,h}

Figures (4)

  • Figure 1: Results for Room. (a) shows the environment. (b) shows the abstracted feature space used for the representation of designed reward functions as a structural constraint. (c) shows results for the setting with a single learner. (d) shows results for the setting with a diverse group of learners with different initial policies. ExpAdaRD designs adaptive reward functions w.r.t. the learner's current policies, whereas other techniques are agnostic to the learner's policy. See Section \ref{['sec:evaluation:envfourroom']} for details.
  • Figure 2: Results for LineK. (a) shows the environment. (b) shows the tree-based feature space used for the representation of designed reward functions as a structural constraint. (c) shows results for the setting with a single learner. (d) shows results for the setting with a diverse group of learners with different initial policies. ExpAdaRD designs adaptive reward functions w.r.t. the learner's current policies, whereas other techniques are agnostic to the learner's policy. See Section \ref{['sec:evaluation:envlinekey']} for details.
  • Figure 3: Visualization of reward functions designed by different techniques in the Room environment for all four actions $\{\textnormal{"up"}, \textnormal{"left"}, \textnormal{"down"}, \textnormal{"right"}\}$. (a) shows original reward function ${R}^{\textsc{Orig}}$. (b) shows reward function ${R}^{\textsc{Invar}}$. (c) shows reward function ${R}^{\textsc{ExpRD}}$ designed by expert-driven non-adaptive reward design technique devidze2021explicable. (d, e, f) show reward functions ${R}^{\textsc{ExpAdaRD}}$ designed by our framework ExpAdaRD for three learners, each with its distinct initial policy, at different training episodes $k$. A negative reward is shown in Red color with the sign "-", a positive reward is shown in Blue color with the sign "+", and a zero reward is shown in white. The color intensity indicates the magnitude of the reward.
  • Figure 4: Visualization of reward functions designed by different techniques in the LineK environment for all three actions $\{\textnormal{"left"}, \textnormal{"right"}, \textnormal{"pick"}\}$. (a) shows original reward function ${R}^{\textsc{Orig}}$. (b) shows reward function ${R}^{\textsc{Invar}}$. (c) shows reward function ${R}^{\textsc{ExpRD}}$ designed by expert-driven non-adaptive reward design technique devidze2021explicable. (d, e, f) show reward functions ${R}^{\textsc{ExpAdaRD}}$ designed by our framework ExpAdaRD for three learners, each with its distinct initial policy, at different training episodes $k$. These plots illustrate reward values for all combinations of triplets: agent's location (indicated as "key loc", "goal loc" in tree plots), agent's status whether it has acquired the key or not (indicated as "has key" in tree plots and letter "K" in bar plots), and three actions (indicated as 'l' for "left", 'r' for "right", 'p' for "pick"). A negative reward is shown in Red color with the sign "-", a positive reward is shown in Blue color with the sign "+", and a zero reward is shown in white. The color intensity indicates the reward magnitude.

Theorems & Definitions (2)

  • Proposition 1
  • Theorem 1