Learning Utilities from Demonstrations in Markov Decision Processes

TL;DR

This work introduces a risk-sensitive framework for learning agents in Markov Decision Processes by modeling the observed risk attitude with a utility function $U^E$ and defining Utility Learning (UL) to infer $U^E$ from demonstrations. It formalizes partial identifiability of $U^E$ and shows that demonstrations across multiple environments can help recover the true risk attitude. The authors propose two provably-efficient online algorithms, CATY-UL and TRACTOR-UL, that operate in a finite-data regime using discretization, an enlarged RS-MDP, and return-distribution projections to identify utilities compatible with observed behavior. They provide theoretical guarantees and validate the approach through proof-of-concept experiments with human data and simulated scenarios. Overall, the paper advances risk-aware imitation learning by separating task reward from risk attitude and offering scalable methods with identifiability considerations.

Abstract

Our goal is to extract useful knowledge from demonstrations of behavior in sequential decision-making problems. Although it is well-known that humans commonly engage in risk-sensitive behaviors in the presence of stochasticity, most Inverse Reinforcement Learning (IRL) models assume a risk-neutral agent. Beyond introducing model misspecification, these models do not directly capture the risk attitude of the observed agent, which can be crucial in many applications. In this paper, we propose a novel model of behavior in Markov Decision Processes (MDPs) that explicitly represents the agent's risk attitude through a utility function. We then define the Utility Learning (UL) problem as the task of inferring the observed agent's risk attitude, encoded via a utility function, from demonstrations in MDPs, and we analyze the partial identifiability of the agent's utility. Furthermore, we devise two provably efficient algorithms for UL in a finite-data regime, and we analyze their sample complexity. We conclude with proof-of-concept experiments that empirically validate both our model and our algorithms.

Figures (32)

• Figure 1: The MDP considered in Example \ref{['example: non markovian policy']}.
• Figure 2: (Left) MDP of Example \ref{['example: UE partially identifiable']}. (Middle) its feasible set with a sample utility $U'$. (Right) plot of $U'$ with linear interpolation.
• Figure 3: Simulations of TRACTOR-UL with various step sizes $\alpha$. The shaded regions are the standard deviation over 5 seeds.
• Figure 4: MDP for the proof of Proposition \ref{['prop: lose in performance with markovianity']}.
• Figure 5: MDP for the proof of Proposition \ref{['prop: policy non stationary']}.

Theorems & Definitions (74)

• Example 3.1
• Proposition 1
• Proposition 2
• Definition 4.1: Utility Learning (UL)
• Example 4.1
• Proposition 3
• Proposition 4
• Proposition 5
• Proposition 6
• Proposition 7