Preference-Based Planning in Stochastic Environments: From Partially-Ordered Temporal Goals to Most Preferred Policies

TL;DR

It is proved that finding a most preferred policy is equivalent to computing a Pareto-optimal policy in a multi-objective MDP that is constructed from the original MDP, the preference automaton, and the chosen stochastic ordering relation.

Abstract

Human preferences are not always represented via complete linear orders: It is natural to employ partially-ordered preferences for expressing incomparable outcomes. In this work, we consider decision-making and probabilistic planning in stochastic systems modeled as Markov decision processes (MDPs), given a partially ordered preference over a set of temporally extended goals. Specifically, each temporally extended goal is expressed using a formula in Linear Temporal Logic on Finite Traces (LTL$_f$). To plan with the partially ordered preference, we introduce order theory to map a preference over temporal goals to a preference over policies for the MDP. Accordingly, a most preferred policy under a stochastic ordering induces a stochastic nondominated probability distribution over the finite paths in the MDP. To synthesize a most preferred policy, our technical approach includes two key steps. In the first step, we develop a procedure to transform a partially ordered preference over temporal goals into a computational model, called preference automaton, which is a semi-automaton with a partial order over acceptance conditions. In the second step, we prove that finding a most preferred policy is equivalent to computing a Pareto-optimal policy in a multi-objective MDP that is constructed from the original MDP, the preference automaton, and the chosen stochastic ordering relation. Throughout the paper, we employ running examples to illustrate the proposed preference specification and solution approaches. We demonstrate the efficacy of our algorithm using these examples, providing detailed analysis, and then discuss several potential future directions.

Figures (5)

• Figure 1: a) Bob's Garden. b) Bob's preferences on how the bee robot should perform the task of pollinating the flowers.
• Figure 2: PDFA for the example in Figure \ref{['fig:gap_garden']}. Left) The DFA structure of PDFA. Right) The preference graph of PDFA.
• Figure 3: PDFA for the example in Figure \ref{['fig:gap_garden']}. Left) Three DFAs for three LTL$_f$ formulas $\varphi_1$, $\varphi_2$, and $\varphi_3$, for which the user preference is: $\varphi_1 \triangleright \varphi_2$, $\varphi_1 \triangleright \varphi_3$, $\varphi_2 \nparallel \varphi_3$. Right) The pdfa constructed by our algorithm for the LTL$_f$ formulas and the preference over them. The output of each state---the set of formulas that satisfies every string that ends at that state---is shown in blue and the most preferred formulas for each state is shown in red.
• Figure 4: a)-d) The DFAs for $p_1-p_4$ for the example Figure \ref{['fig:gap_garden']}, which are constrcuted by our online tools, available at https://akulkarni.me/prefltlf2pdfa.html. e)-f) The PDFA for the example in Figure \ref{['fig:gap_garden']}, which is constructed the implementation of our algorithm for converting a preference model over ltlf formulas into a pdfa.
• Figure 5: The probabilities of satisfying objectives $\{p_1, p_2\}$ and $\{p_1, p_3\}$ by the computed policies for weak-stochastic ordering who satisfy the objective $\{p_1\}$ with probability $0.2406$.

Theorems & Definitions (36)

• Definition 1: tlmdp
• Definition 2: Syntax of ltlf de2013linear
• Example 1
• Definition 3: dfa
• Definition 4
• Definition 5
• Definition 6
• Example 2
• Definition 7
• Definition 8