Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Peer-Aware Cost Estimation in Nonlinear General-Sum Dynamic Games for Mutual Learning and Intent Inference

Seyed Yousef Soltanian, Wenlong Zhang

TL;DR

This work tackles incomplete-information, nonlinear general-sum dynamic games by introducing N-PACE, a peer-aware cost estimation framework built on iterative linear-quadratic (ILQ) approximations. Each agent jointly infers the peer's objective while updating its own policy, explicitly modeling the peer's learning dynamics to avoid bias from treating the peer as an expert. The approach enables intent communication by embedding an signaling term that guides mutual learning, and it is validated across lunar lander, lane merging, and intersection-driving scenarios, showing improved safety, convergence, and coordination with real-time computational performance. The results highlight the practical impact of accounting for mutual learning in multi-agent autonomous systems and point to future work on learning the learning dynamics and human-robot interaction applications.

Abstract

Dynamic game theory is a powerful tool in modeling multi-agent interactions and human-robot systems. In practice, since the objective functions of both agents may not be explicitly known to each other, these interactions can be modeled as incomplete-information general-sum dynamic games. Solving for equilibrium policies for such games presents a major challenge, especially if the games involve nonlinear underlying dynamics. To simplify the problem, existing work often assumes that one agent is an expert with complete information about its peer, which can lead to biased estimates and failures in coordination. To address this challenge, we propose a nonlinear peer-aware cost estimation (N-PACE) algorithm for general-sum dynamic games. In N-PACE, using iterative linear quadratic (ILQ) approximation of dynamic games, each agent explicitly models the learning dynamics of its peer agent while inferring their objective functions and updating its own control policy accordingly in real time, which leads to unbiased and fast learning of the unknown objective function of the peer agent. Additionally, we demonstrate how N-PACE enables intent communication by explicitly modeling the peer's learning dynamics. Finally, we show how N-PACE outperforms baseline methods that disregard the learning behavior of the other agent, both analytically and using our case studies

Peer-Aware Cost Estimation in Nonlinear General-Sum Dynamic Games for Mutual Learning and Intent Inference

TL;DR

This work tackles incomplete-information, nonlinear general-sum dynamic games by introducing N-PACE, a peer-aware cost estimation framework built on iterative linear-quadratic (ILQ) approximations. Each agent jointly infers the peer's objective while updating its own policy, explicitly modeling the peer's learning dynamics to avoid bias from treating the peer as an expert. The approach enables intent communication by embedding an signaling term that guides mutual learning, and it is validated across lunar lander, lane merging, and intersection-driving scenarios, showing improved safety, convergence, and coordination with real-time computational performance. The results highlight the practical impact of accounting for mutual learning in multi-agent autonomous systems and point to future work on learning the learning dynamics and human-robot interaction applications.

Abstract

Dynamic game theory is a powerful tool in modeling multi-agent interactions and human-robot systems. In practice, since the objective functions of both agents may not be explicitly known to each other, these interactions can be modeled as incomplete-information general-sum dynamic games. Solving for equilibrium policies for such games presents a major challenge, especially if the games involve nonlinear underlying dynamics. To simplify the problem, existing work often assumes that one agent is an expert with complete information about its peer, which can lead to biased estimates and failures in coordination. To address this challenge, we propose a nonlinear peer-aware cost estimation (N-PACE) algorithm for general-sum dynamic games. In N-PACE, using iterative linear quadratic (ILQ) approximation of dynamic games, each agent explicitly models the learning dynamics of its peer agent while inferring their objective functions and updating its own control policy accordingly in real time, which leads to unbiased and fast learning of the unknown objective function of the peer agent. Additionally, we demonstrate how N-PACE enables intent communication by explicitly modeling the peer's learning dynamics. Finally, we show how N-PACE outperforms baseline methods that disregard the learning behavior of the other agent, both analytically and using our case studies

Paper Structure

This paper contains 18 sections, 3 theorems, 17 equations, 6 figures, 3 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Related Work
  3. Problem Formulation
  4. ILQGame for Multi-Agent Interactions
  5. Modeling the Learning Dynamics
  6. N-PACE for Multi-Agent Interactions
  7. N-PACE for Intent Communication
  8. Case Studies
  9. Case Study 1: Assistive Lunar Lander
  10. Case Study 2: Lane Merging
  11. Case Study 3: Interaction Driving
  12. CONCLUSIONS AND FUTURE WORK
  13. Proofs
  14. Details of Bayesian Gradient Descent Learning Function
  15. Details of The Non Game Theoretic MPC in Case Study 1
  16. ...and 3 more sections

Key Result

Proposition 1

Suppose the stage costs $g_t^k(s_t,a_t^i,a_t^j;\theta^k)$ in eq:J have Hessians with respect to $(s,a^i,a^j)$ that are independent of $\theta^k$, with $\theta^k$ entering only linearly in the first-order terms, and on any compact operating set $s_t \in \Omega$, the policy of each agent admits the fo

Figures (6)

  • Figure 1: Demonstration of the core idea in N-PACE using a simulated general-sum game between two autonomous cars at an intersection. In (a), each agent assumes its peer knows its aggressiveness level, leading to biased inferences and possible collisions. In (b), N-PACE models the peer agent as a learning entity, resulting in a safe intersection crossing.
  • Figure 2: Results of Case Study 1, the assistive lunar lander. (a) Trajectory comparison between baselines and N-PACE. (b) and (c) show each agent’s learning performance under different algorithms (d) comparison of final landing position error. (e) Final sum of intent learning error. (f) and (g) cumulative cost comparisons under different approaches for each agent.
  • Figure 3: Trajectories of the lane-merging case study, the trajectories are plotted relative to the blue car coordinate (a) Safe lane merging under complete information (b) baseline methods, a risky behavior in merging from the aggressive driver (red). (c)Non-risky lane merging in N-PACE similar to the complete game, despite a 30% error in modeling the peer's initial estimate.
  • Figure 4: a) & b) Robustness of N-PACE in intent inference under mismatched assumptions about the peer’s initial belief. c) & d) robustness to wrong choice of initial variance in peer's GD-based Bayesian inference. Envelopes representing 50 mismatched runs.
  • Figure 5: Comparison of $\log(1+\text{MSE})$ learning error across 300 Monte Carlo simulations of case study 3 with random values of aggressiveness.
  • ...and 1 more figures

Theorems & Definitions (7)

  • Remark 1
  • Remark 2
  • Remark 3
  • Remark 4
  • Proposition 1: N-PACE v.s treating the peer as an expert
  • Lemma 1: Monotone reduction of intent estimation error under intent communication
  • Corollary 1: Intent communication never hurts