Reward Distance Comparisons Under Transition Sparsity

TL;DR

This work tackles the challenge of comparing reward functions without relying on policy learning by introducing the Sparsity Resilient Reward Distance (SRRD), a direct reward comparison pseudometric designed for highly sparse transition data. SRRD blends canonicalization concepts from existing methods with additional reward-expectation terms that leverage observed sample distributions, achieving policy-invariant comparisons even when full transition coverage is unavailable. The authors provide theoretical robustness results via Relative Shaping Errors and a regret-bound framework, and empirically validate SRRD against EPIC and DARD across Gridworld, Bouncing Balls, Drone Combat, StarCraft II, Robomimic, Montezuma’s Revenge, StarCraft II, and MIMIC-IV domains. They demonstrate SRRD’s superior performance under transition sparsity and its effectiveness as a distance measure for classifying agent behaviors through IRL-derived rewards, with potential to accelerate IRL workflows and improve reward evaluation. The work points to future extensions that address non-potential shaping, scaling to neural-reward representations, and multicriteria invariance to support broader applicability in real-world reward modeling tasks.

Abstract

Reward comparisons are vital for evaluating differences in agent behaviors induced by a set of reward functions. Most conventional techniques utilize the input reward functions to learn optimized policies, which are then used to compare agent behaviors. However, learning these policies can be computationally expensive and can also raise safety concerns. Direct reward comparison techniques obviate policy learning but suffer from transition sparsity, where only a small subset of transitions are sampled due to data collection challenges and feasibility constraints. Existing state-of-the-art direct reward comparison methods are ill-suited for these sparse conditions since they require high transition coverage, where the majority of transitions from a given coverage distribution are sampled. When this requirement is not satisfied, a distribution mismatch between sampled and expected transitions can occur, leading to significant errors. This paper introduces the Sparsity Resilient Reward Distance (SRRD) pseudometric, designed to eliminate the need for high transition coverage by accommodating diverse sample distributions, which are common under transition sparsity. We provide theoretical justification for SRRD's robustness and conduct experiments to demonstrate its practical efficacy across multiple domains.

Figures (13)

• Figure 1: (Transition Sparsity in a $10 \times 10$ Gridworld Domain) In this illustration, each transition starts from a starting state $s$ and ends in a destination state $s'$. For clarity in visualization, we consider action-independent rewards, $R(s, s')$. In (a), high transition coverage results from a high rollout count (number of policy rollouts) in the absence of feasibility constraints, leading to the majority of transitions being sampled (blue points). In (b), low coverage results from a low rollout count in the absence of feasibility constraints, leading to fewer sampled transitions (red points). In (c), low coverage results from feasibility constraints, such as movement restrictions that only allow actions to adjacent cells, which can significantly reduce the space of sampled transitions (green points) irrespective of rollout count.
• Figure 2: (Impact of unsampled transitions on canonicalizing $R(s_1, a_1, s_2)$) Sampled transitions are those explored in the reward sample, while expected transitions are those anticipated by $\hat{C}_{EPIC}$ assuming full coverage. As coverage decreases from (a) to (c), due to a reduction in the number of sampled transitions, the standard deviation of $\hat{C}_{EPIC}(R)(s_1, a, s_2)$ increases, indicating $\hat{C}_{EPIC}$'s increased instability to unsampled transitions. For comparison, $\hat{C}_{SRRD}$ and $\hat{C}_{DARD}$ have lower standard deviations, signifying higher stability.
• Figure 3: (Transition Sparsity). The figure illustrates the performance of reward comparison pseudometrics in identifying the similarity between potentially shaped reward functions under two conditions: (a) limited sampling and (b) feasibility constraints. A more accurate pseudometric yields a Pearson distance $D_\rho$ close to $0$, indicating a high degree of similarity between shaped reward functions, while a less accurate pseudometric results in $D_\rho$ close to $1$. In both experiments, transition coverage is calculated as the ratio of sampled transitions to the set of all theoretically possible transitions $|S \times A \times S|$, including both feasible and unfeasible transitions. Each coverage data point represents an average over $200$ simulations at a constant policy rollout count, with coverage data points generated by varying the number of policy rollouts from $1$ to $2000$ (see Appendix \ref{['ap:Appendix_C.1.4']}). In panel (a), EPIC and DARD lag behind SRRD at low transition coverage due to limited sampling, but their performance gradually improves as coverage increases with higher rollout counts. In panel (b), movement restrictions significantly reduce transition coverage, regardless of rollout sampling frequency, which negatively impacts EPIC's performance (almost similar to DIRECT).
• Figure 4: A transition graph with $10$ states $\{x_0, ...x_9\}$, and a single action $\{a_1\}$. State subsets are defined based on the transition: $(x_0, a_1, x_1)$.
• Figure 5: (Non-Potential Shaping Effects): As the severity of randomly generated noise increases from part (a) to (c), rewards deviate more from potential shaping, hence, all the pseudometrics degrade in performance. In the end (part (c)), the pseudometrics perform similarly to DIRECT, showing that canonicalization does not yield any additional advantages when the rewards significantly deviate from potential shaping.

Theorems & Definitions (31)

• Definition 1: Sparsity Resilient Canonically Shaped Reward
• Proposition 1
• Definition 2
• Definition 3
• Definition 4: Forward transitions
• Definition 5: Non-forward transitions
• Theorem 1
• proof
• Definition 6
• Definition 7