Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Inverse Reinforcement Learning with Sub-optimal Experts

Riccardo Poiani, Gabriele Curti, Alberto Maria Metelli, Marcello Restelli

TL;DR

This work addresses the ill-posed nature of Inverse Reinforcement Learning when demonstrations come from multiple sub-optimal experts. It introduces IRL-SE, formalizing the feasible reward set $\mathcal{R}_{\bar{\mathfrak{B}}}$ and providing both implicit and explicit characterizations that show sub-optimal experts can substantially shrink the set of compatible rewards. The authors develop a PAC-based learning framework and derive lower bounds on the sample complexity, then propose a uniform-sampling algorithm US-IRL-SE that achieves minimax-optimal guarantees when the sub-optimality gaps $\xi_i$ are small. The results highlight a fundamental trade-off: leveraging sub-optimal demonstrations reduces reward ambiguity but increases statistical complexity, guiding design of data collection strategies for IRL in realistic, multi-expert settings. These insights offer a principled path toward more identifiable reward functions in human-in-the-loop and other multi-expert contexts.

Abstract

Inverse Reinforcement Learning (IRL) techniques deal with the problem of deducing a reward function that explains the behavior of an expert agent who is assumed to act optimally in an underlying unknown task. In several problems of interest, however, it is possible to observe the behavior of multiple experts with different degree of optimality (e.g., racing drivers whose skills ranges from amateurs to professionals). For this reason, in this work, we extend the IRL formulation to problems where, in addition to demonstrations from the optimal agent, we can observe the behavior of multiple sub-optimal experts. Given this problem, we first study the theoretical properties of the class of reward functions that are compatible with a given set of experts, i.e., the feasible reward set. Our results show that the presence of multiple sub-optimal experts can significantly shrink the set of compatible rewards. Furthermore, we study the statistical complexity of estimating the feasible reward set with a generative model. To this end, we analyze a uniform sampling algorithm that results in being minimax optimal whenever the sub-optimal experts' performance level is sufficiently close to the one of the optimal agent.

Inverse Reinforcement Learning with Sub-optimal Experts

TL;DR

This work addresses the ill-posed nature of Inverse Reinforcement Learning when demonstrations come from multiple sub-optimal experts. It introduces IRL-SE, formalizing the feasible reward set and providing both implicit and explicit characterizations that show sub-optimal experts can substantially shrink the set of compatible rewards. The authors develop a PAC-based learning framework and derive lower bounds on the sample complexity, then propose a uniform-sampling algorithm US-IRL-SE that achieves minimax-optimal guarantees when the sub-optimality gaps are small. The results highlight a fundamental trade-off: leveraging sub-optimal demonstrations reduces reward ambiguity but increases statistical complexity, guiding design of data collection strategies for IRL in realistic, multi-expert settings. These insights offer a principled path toward more identifiable reward functions in human-in-the-loop and other multi-expert contexts.

Abstract

Inverse Reinforcement Learning (IRL) techniques deal with the problem of deducing a reward function that explains the behavior of an expert agent who is assumed to act optimally in an underlying unknown task. In several problems of interest, however, it is possible to observe the behavior of multiple experts with different degree of optimality (e.g., racing drivers whose skills ranges from amateurs to professionals). For this reason, in this work, we extend the IRL formulation to problems where, in addition to demonstrations from the optimal agent, we can observe the behavior of multiple sub-optimal experts. Given this problem, we first study the theoretical properties of the class of reward functions that are compatible with a given set of experts, i.e., the feasible reward set. Our results show that the presence of multiple sub-optimal experts can significantly shrink the set of compatible rewards. Furthermore, we study the statistical complexity of estimating the feasible reward set with a generative model. To this end, we analyze a uniform sampling algorithm that results in being minimax optimal whenever the sub-optimal experts' performance level is sufficiently close to the one of the optimal agent.
Paper Structure (49 sections, 24 theorems, 157 equations, 4 figures, 2 tables, 1 algorithm)

This paper contains 49 sections, 24 theorems, 157 equations, 4 figures, 2 tables, 1 algorithm.

Table of Contents

  1. INTRODUCTION
  2. Contributions and Outline
  3. PRELIMINARIES
  4. Notation
  5. Markov Decision Processes
  6. Operators
  7. Value Functions and Optimality
  8. Inverse Reinforcement Learning
  9. IRL with Sub-optimal Experts
  10. Empirical Estimates
  11. SUB-OPTIMAL EXPERTS AND THE FEASIBLE REWARD SET
  12. Implicit Formulation of $\mathcal{R}_{\bar{\mathfrak{B}}}$
  13. Explicit Formulation of $\mathcal{R}_{\bar{\mathfrak{B}}}$
  14. LEARNING THE FEASIBLE SET
  15. PAC Framework
  16. ...and 34 more sections

Key Result

Lemma 1

Let $\bar{\mathfrak{B}}$ be an IRL problem with sub-optimal experts. Let $r \in [0,1]^{\mathcal{S} \times \mathcal{A}}$. Then, $r \in \mathcal{R}_{\bar{\mathfrak{B}}}$ if and only if the following conditions are satisfied:

Figures (4)

  • Figure 1: MDP example, with $2$ states and $2$ experts, that highlights the benefits of sub-optimal agents (Example \ref{['exe:2']}). In $S_{1}$ both $\pi_{E_1}$ and $\pi_{E_i}$ are identical, i.e., $\pi_{E_1}(\bar{A}|S_1)=\pi_{E_i}(\bar{A}|S_1)=1$.
  • Figure 2: Representation of the IRL-SE problem for the instances used in Theorem \ref{['theo:lb-opt-exp-caso1']}.
  • Figure 3: Representation of the IRL-SE problem for the instances used in Theorem \ref{['theo:lb-opt-exp-caso2']}.
  • Figure 4: Representation of the IRL-SE problem for the instances used in Theorem \ref{['theo:lb-sub-opt']}.

Theorems & Definitions (40)

  • Lemma 1
  • Example 1
  • Example 2
  • Theorem 3
  • Example 4
  • Theorem 5
  • Theorem 6
  • Lemma 2
  • Lemma 3
  • Lemma 4
  • ...and 30 more