Trajectory Modeling via Random Utility Inverse Reinforcement Learning
Anselmo R. Pitombeira-Neto, Helano P. Santos, Ticiana L. Coelho da Silva, José Antonio F. de Macedo
TL;DR
The paper addresses modeling driver trajectories observed by sparse road-network sensors by introducing random utility inverse reinforcement learning (RU-IRL), which assumes deterministic optimal behavior under an unknown reward with unobserved utility $\epsilon$. It derives a Markov decision process with extended state $(s,\epsilon)$, proves existence of the value function $v_{\bm{\theta}}$ via contraction (for $0<\gamma<1$) and shows ME-IRL is a special case at $\gamma=1$, enabling exact normalization without enumerating trajectories. Parameters are estimated with Bayesian inference using Metropolis-Hastings, exploiting a scale invariance that makes only ratios $\beta_k/\alpha$ identifiable. A case study on Fortaleza data demonstrates RU-IRL can recover the relative importance of distance versus travel time and yield competitive next-location predictions with far fewer parameters than a Markov model. The approach offers a transparent, statistically principled alternative to black-box trajectory models and can be extended to capture user heterogeneity and sensor noise.
Abstract
We consider the problem of modeling trajectories of drivers in a road network from the perspective of inverse reinforcement learning. Cars are detected by sensors placed on sparsely distributed points on the street network of a city. As rational agents, drivers are trying to maximize some reward function unknown to an external observer. We apply the concept of random utility from econometrics to model the unknown reward function as a function of observed and unobserved features. In contrast to current inverse reinforcement learning approaches, we do not assume that agents act according to a stochastic policy; rather, we assume that agents act according to a deterministic optimal policy and show that randomness in data arises because the exact rewards are not fully observed by an external observer. We introduce the concept of extended state to cope with unobserved features and develop a Markov decision process formulation of drivers decisions. We present theoretical results which guarantee the existence of solutions and show that maximum entropy inverse reinforcement learning is a particular case of our approach. Finally, we illustrate Bayesian inference on model parameters through a case study with real trajectory data from a large city in Brazil.