Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Sequential Decision Making in Stochastic Games with Incomplete Preferences over Temporal Objectives

Abhishek Ninad Kulkarni, Jie Fu, Ufuk Topcu

TL;DR

The paper addresses strategy synthesis in two-player stochastic games with adversarial and potentially incomplete preferences over temporal objectives expressed as $LTL_f$ formulas. It introduces non-dominated almost-sure winning (ndaswin) strategies and a scalar rank metric based on undominance to compare outcomes without requiring full multi-utility representations. Using an automata-theoretic product construction, it reduces preference comparisons to state-orderings in a product game and proves that ndaswin strategies yield a qualitative Nash equilibrium. A drone-delivery case study demonstrates practical applicability, showing how incomparabilities are handled and how robust, preference-aligned behavior can be achieved in adversarial, stochastic settings.

Abstract

Ensuring that AI systems make strategic decisions aligned with the specified preferences in adversarial sequential interactions is a critical challenge for developing trustworthy AI systems, especially when the environment is stochastic and players' incomplete preferences leave some outcomes unranked. We study the problem of synthesizing preference-satisfying strategies in two-player stochastic games on graphs where players have opposite (possibly incomplete) preferences over a set of temporal goals. We represent these goals using linear temporal logic over finite traces (LTLf), which enables modeling the nuances of human preferences where temporal goals need not be mutually exclusive and comparison between some goals may be unspecified. We introduce a solution concept of non-dominated almost-sure winning, which guarantees to achieve a most preferred outcome aligned with specified preferences while maintaining robustness against the adversarial behaviors of the opponent. Our results show that strategy profiles based on this concept are Nash equilibria in the game where players are risk-averse, thus providing a practical framework for evaluating and ensuring stable, preference-aligned outcomes in the game. Using a drone delivery example, we demonstrate that our contributions offer valuable insights not only for synthesizing rational behavior under incomplete preferences but also for designing games that motivate the desired behavior from the players in adversarial conditions.

Sequential Decision Making in Stochastic Games with Incomplete Preferences over Temporal Objectives

TL;DR

The paper addresses strategy synthesis in two-player stochastic games with adversarial and potentially incomplete preferences over temporal objectives expressed as formulas. It introduces non-dominated almost-sure winning (ndaswin) strategies and a scalar rank metric based on undominance to compare outcomes without requiring full multi-utility representations. Using an automata-theoretic product construction, it reduces preference comparisons to state-orderings in a product game and proves that ndaswin strategies yield a qualitative Nash equilibrium. A drone-delivery case study demonstrates practical applicability, showing how incomparabilities are handled and how robust, preference-aligned behavior can be achieved in adversarial, stochastic settings.

Abstract

Ensuring that AI systems make strategic decisions aligned with the specified preferences in adversarial sequential interactions is a critical challenge for developing trustworthy AI systems, especially when the environment is stochastic and players' incomplete preferences leave some outcomes unranked. We study the problem of synthesizing preference-satisfying strategies in two-player stochastic games on graphs where players have opposite (possibly incomplete) preferences over a set of temporal goals. We represent these goals using linear temporal logic over finite traces (LTLf), which enables modeling the nuances of human preferences where temporal goals need not be mutually exclusive and comparison between some goals may be unspecified. We introduce a solution concept of non-dominated almost-sure winning, which guarantees to achieve a most preferred outcome aligned with specified preferences while maintaining robustness against the adversarial behaviors of the opponent. Our results show that strategy profiles based on this concept are Nash equilibria in the game where players are risk-averse, thus providing a practical framework for evaluating and ensuring stable, preference-aligned outcomes in the game. Using a drone delivery example, we demonstrate that our contributions offer valuable insights not only for synthesizing rational behavior under incomplete preferences but also for designing games that motivate the desired behavior from the players in adversarial conditions.

Paper Structure

This paper contains 14 sections, 9 theorems, 4 equations, 3 figures, 1 algorithm.

Table of Contents

  1. Introduction
  2. Preliminaries
  3. Game Model
  4. Specifying Temporal Goals
  5. Automata-theoretic Planning with ltlf
  6. Preference Modeling
  7. Problem Formulation
  8. Main Results
  9. Product Game
  10. Ranks: A Measure of Quality of Outcome
  11. Synthesis of Non-dominated Almost-sure Winning Strategy
  12. Qualitative Nash Equilibrium
  13. Experiment
  14. Conclusion

Key Result

Proposition 1

Given any finite paths $\rho, \rho'$ in $G$, let $\varrho, \varrho'$ be their traces in $H$. Then, $L(\rho) \succeq_1 L(\rho')$ holds if and only if $\mathsf{Last}(\varrho) \succeq_{\mathcal{E}_1} \mathsf{Last}(\varrho')$.

Figures (3)

  • Figure 1: Drone delivery in hostile environment.
  • Figure 2: Preference automaton for the relation defined by $\varphi_1 \triangleright \varphi_2$, $\varphi_1 \triangleright \varphi_3$, $\varphi_4 \triangleright \varphi_2$, and $\varphi_4 \triangleright \varphi_3$.
  • Figure 3: The smallest rank achievable by drone A by following a ndaswin strategy.

Theorems & Definitions (27)

  • Definition 1
  • Definition 2: Proper Strategy
  • Definition 3
  • Definition 4
  • Definition 5
  • Definition 6: Dominating Strategy Profile
  • Definition 7: ndaswin
  • Definition 8
  • Proposition 1
  • proof
  • ...and 17 more