Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Learning the Value Systems of Agents with Preference-based and Inverse Reinforcement Learning

Andrés Holgado-Sánchez, Holger Billhardt, Alberto Fernández, Sascha Ossowski

TL;DR

The paper addresses learning explicit value-grounding and value-system representations to enable value-aligned agents within Agreement Technologies. It formalizes values as a grounded set $V$ with per-value rewards ${R}_{v_i}$ implemented by a grounding function $G_V$, and models decision-making with a Markov Value Decision Process (MVDP) where an agent's value system is a linear aggregation $R_j=f_j({oldsymbol{R}}_V)$. It introduces two learning pipelines: Value Grounding Learning, which uses Preference-based Reinforcement Learning to recover ${R}_{v_i}$ from trajectory comparisons, and Value System Identification, which uses Deep MaxEnt IRL to infer aggregation weights $W_j$ so that $R_j=W_joldsymbol{R}_V$ reproduces observed behavior. The framework is validated in Firefighters and Roadworld, showing accurate grounding recovery and high fidelity in identifying agents' value systems, with strong generalization to unseen trajectories. The work demonstrates that combining PbRL and IRL yields interpretable, context-aware value-aligned agents, while noting limitations and proposing directions for extending groundings and linear value systems to more complex, context-sensitive settings.

Abstract

Agreement Technologies refer to open computer systems in which autonomous software agents interact with one another, typically on behalf of humans, in order to come to mutually acceptable agreements. With the advance of AI systems in recent years, it has become apparent that such agreements, in order to be acceptable to the involved parties, must remain aligned with ethical principles and moral values. However, this is notoriously difficult to ensure, especially as different human users (and their software agents) may hold different value systems, i.e. they may differently weigh the importance of individual moral values. Furthermore, it is often hard to specify the precise meaning of a value in a particular context in a computational manner. Methods to estimate value systems based on human-engineered specifications, e.g. based on value surveys, are limited in scale due to the need for intense human moderation. In this article, we propose a novel method to automatically \emph{learn} value systems from observations and human demonstrations. In particular, we propose a formal model of the \emph{value system learning} problem, its instantiation to sequential decision-making domains based on multi-objective Markov decision processes, as well as tailored preference-based and inverse reinforcement learning algorithms to infer value grounding functions and value systems. The approach is illustrated and evaluated by two simulated use cases.

Learning the Value Systems of Agents with Preference-based and Inverse Reinforcement Learning

TL;DR

The paper addresses learning explicit value-grounding and value-system representations to enable value-aligned agents within Agreement Technologies. It formalizes values as a grounded set with per-value rewards implemented by a grounding function , and models decision-making with a Markov Value Decision Process (MVDP) where an agent's value system is a linear aggregation . It introduces two learning pipelines: Value Grounding Learning, which uses Preference-based Reinforcement Learning to recover from trajectory comparisons, and Value System Identification, which uses Deep MaxEnt IRL to infer aggregation weights so that reproduces observed behavior. The framework is validated in Firefighters and Roadworld, showing accurate grounding recovery and high fidelity in identifying agents' value systems, with strong generalization to unseen trajectories. The work demonstrates that combining PbRL and IRL yields interpretable, context-aware value-aligned agents, while noting limitations and proposing directions for extending groundings and linear value systems to more complex, context-sensitive settings.

Abstract

Agreement Technologies refer to open computer systems in which autonomous software agents interact with one another, typically on behalf of humans, in order to come to mutually acceptable agreements. With the advance of AI systems in recent years, it has become apparent that such agreements, in order to be acceptable to the involved parties, must remain aligned with ethical principles and moral values. However, this is notoriously difficult to ensure, especially as different human users (and their software agents) may hold different value systems, i.e. they may differently weigh the importance of individual moral values. Furthermore, it is often hard to specify the precise meaning of a value in a particular context in a computational manner. Methods to estimate value systems based on human-engineered specifications, e.g. based on value surveys, are limited in scale due to the need for intense human moderation. In this article, we propose a novel method to automatically \emph{learn} value systems from observations and human demonstrations. In particular, we propose a formal model of the \emph{value system learning} problem, its instantiation to sequential decision-making domains based on multi-objective Markov decision processes, as well as tailored preference-based and inverse reinforcement learning algorithms to infer value grounding functions and value systems. The approach is illustrated and evaluated by two simulated use cases.
Paper Structure (26 sections, 3 theorems, 29 equations, 5 figures, 12 tables, 2 algorithms)

This paper contains 26 sections, 3 theorems, 29 equations, 5 figures, 12 tables, 2 algorithms.

Table of Contents

  1. Introduction
  2. Background
  3. Agreement Technologies
  4. Value awareness and ethical decision-making
  5. Learning value specifications from demonstrations
  6. Inverse and Preference-based Reinforcement Learning
  7. Value System Learning
  8. Value system representation model and learning framework
  9. Value system learning in sequential decisions
  10. Algorithms for Value System Learning in MVDP
  11. Value Grounding Learning
  12. Value System Identification
  13. Evaluation
  14. Use cases description
  15. Firefighters
  16. ...and 11 more sections

Key Result

Proposition 1

Let $R_{v_i}^{\theta}: S \times A \to \mathbb{R}$ and $\hat{R}_{v_i}^{\hat{\theta}}: S \times A \to \mathbb{R}$ be two reward functions (for value $v_i$) that are parametrized with parameters $\theta,\hat{\theta} \in \Theta$ ($\Theta$ denotes the parameter space). Let $D$ a (finite) dataset of prefe Assume that $\hat{\theta}$ minimizes $L$ in equation eq:groundingloss, i.e. $\hat{\theta} = \mathop

Figures (5)

  • Figure 1: Value system representation model (left side) and its instantiation as a value learning problem in sequential decision-making (right side)
  • Figure 2: Image depicting the Shanghai road network used for this paper (taken from zhao2023routesairl). The green point (upper right part of the figure) is an example origin, and the red point (bottom left) is the destination. The optimal route from the origin to the destination regarding comfort is depicted in blue; the optimal route for sustainability in green; and the optimal route for efficiency in red.
  • Figure 3: Learned value rewards ($\hat{R}_{x}$) versus ground truth rewards ($R_{x}$) for each state-action pair. LEFT: for the two values professionalism (pf) and proximity (px) in the firefighter environment. RIGHT: for the three values sustainability (su), comfort (co) and efficiency (ef) in the Roadworld scenario
  • Figure 4: Average total absolute difference in visitation counts (TVC) per training iteration in the Firefighters scenario when performing value system identification assuming the original reward functions $R_{pf}$ and $R_{px}$ (left) and the learned functions (right). In the legend, "Last" indicates the TVC obtained in the final training iteration, rounded to 2 decimal places
  • Figure 5: Total absolute difference in visitation counts (TVC) per training iteration in the Roadworld scenario when performing value system identification assuming the original reward functions $R_{su}$, $R_{co}$ and $R_{ef}$ (left) and the learned functions (right). Plotted is the average TVC over 10 repetitions with standard deviations. In the legend, "Last" indicates the TVC obtained in the final training iteration, which is exactly $0.0$ in al cases.

Theorems & Definitions (16)

  • Definition 1: Value Alignment Function
  • Definition 2: Grounding function
  • Definition 3: Value system
  • Definition 4: Value System Function
  • Definition 5: Value System function equivalence
  • Definition 6: Markov Value Decision Process
  • Definition 7: Grounding function equivalence
  • Proposition 1
  • proof
  • Proposition 2
  • ...and 6 more