Multi-Objective Planning with Contextual Lexicographic Reward Preferences

TL;DR

This work addresses planning under multiple objectives when objective priorities depend on context. It introduces Contextual Lexicographic MDP (CLMDP), which assigns a context-specific lexicographic ordering and rewards to states, maps states to contexts, and merges per-context policies into a single, conflict-free global policy. A conflict-detection and resolution mechanism guarantees cycle-free execution, and a Bayesian method learns the state-context mapping from limited expert trajectories when it is unknown. The approach is validated in simulation across three domains and on a mobile robot, showing robust balance of objectives and reliable goal achievement under multiple contexts, with theoretical guarantees on correctness and complexity. The results highlight the practicality of context-aware, lexicographic planning in real-world autonomous systems with context-dependent preferences.

Abstract

Autonomous agents are often required to plan under multiple objectives whose preference ordering varies based on context. The agent may encounter multiple contexts during its course of operation, each imposing a distinct lexicographic ordering over the objectives, with potentially different reward functions associated with each context. Existing approaches to multi-objective planning typically consider a single preference ordering over the objectives, across the state space, and do not support planning under multiple objective orderings within an environment. We present Contextual Lexicographic Markov Decision Process (CLMDP), a framework that enables planning under varying lexicographic objective orderings, depending on the context. In a CLMDP, both the objective ordering at a state and the associated reward functions are determined by the context. We employ a Bayesian approach to infer a state-context mapping from expert trajectories. Our algorithm to solve a CLMDP first computes a policy for each objective ordering and then combines them into a single context-aware policy that is valid and cycle-free. The effectiveness of the proposed approach is evaluated in simulation and using a mobile robot.

Figures (5)

• Figure 1: Example of a car navigating in a city environment with three different contexts (urban, highway, and construction), each imposing a unique ordering over the objectives.
• Figure 2: Overview of our solution approach for contextual planning. First, policies are calculated for each context in isolation, across the entire state space, and then compiled into a global policy $\pi_G$ by mapping actions to states based on each state's associated context. Second, $\pi_G$ is analyzed for cycles by estimating goal reachability from each state. Finally, the detected conflicts are resolved by updating lower priority context policies conditioned on fixed actions of higher priority contexts.
• Figure 3: Performance in all objectives ($o_1,\,o_2,\,o_3$) normalized against maximum reward, averaged over 100 trials across 5 instances of each domain, represented as a heatmap with darker shade denoting higher and lighter denoting lower reward.
• Figure 4: Minimum objective value from each technique averaged over 100 trials in five instances of each domain.
• Figure 5: Comparison of the paths taken by our warehouse agent during delivery task in our indoor setup.

Theorems & Definitions (7)

• Definition 1
• Definition 2
• Definition 3
• proposition 1
• proposition 2
• proposition 3
• proposition 4