Reach-avoid semi-Markov decision processes with time-varying obstacles
Yanyun Li, Xianping Guo
TL;DR
The paper addresses the challenge of computing maximal reach-avoid probabilities in finite-horizon semi-Markov decision processes with time-varying obstacles, which induce non-homogeneous dynamics. It introduces a state-space augmentation that enlarges the process to a two-dimensional model and proves an equivalence with the original problem, enabling the use of a homogeneous framework. An improved value-type algorithm is developed for the augmented model, with convergence guarantees and a clear procedure to obtain ε-optimal policies and translate them back to the original semi-MDP. A plane-flight example demonstrates the practical computation and shows how altering obstacle timing can affect reach-avoid performance, highlighting the method's potential for planning under time-varying constraints.
Abstract
We consider the maximal reach-avoid probability to a target in finite horizon for semi-Markov decision processes with time-varying obstacles. Since the variance of the obstacle set, the model \eqref{Model} is non-homogeneous. To overcome such difficulty, we construct a related two-dimensional model \eqref{newModel}, and then prove the equivalence between such reach-avoid probability of the original model and that of the related two-dimensional one. For the related two-dimensional model, we analyze some special characteristics of the equivalent reach-avoid probability. On this basis, we provide a special improved value-type algorithm to obtain the equivalent maximal reach-avoid probability and its $ε$-optimal policy. Then, at the last step of the algorithm, by the equivalence between these two models, we obtain the original maximal reach-avoid probability and its $ε$-optimal policy for the original model.