HAVA: Hybrid Approach to Value-Alignment through Reward Weighing for Reinforcement Learning

TL;DR

This work addresses value alignment for reinforcement learning by merging mandatory rule-based norms with learned social norms within a unified framework called HAVA. HAVA augments the MDP with a reputation variable and uses it to weight task rewards, ensuring actions comply with safety/legal norms while encouraging socially acceptable behavior. Through grid-world and SUMO-based junction experiments, HAVA demonstrates that combining RB and DD yields value-aligned policies, avoiding both unsafe norm violations and socially inappropriate strategies observed in purely rule-based or purely data-driven approaches. The results highlight the importance of norm severity awareness and forgiveness dynamics in shaping policies that resemble human behavior and can generalize to complex, continuous environments.

Abstract

Our society is governed by a set of norms which together bring about the values we cherish such as safety, fairness or trustworthiness. The goal of value-alignment is to create agents that not only do their tasks but through their behaviours also promote these values. Many of the norms are written as laws or rules (legal / safety norms) but even more remain unwritten (social norms). Furthermore, the techniques used to represent these norms also differ. Safety / legal norms are often represented explicitly, for example, in some logical language while social norms are typically learned and remain hidden in the parameter space of a neural network. There is a lack of approaches in the literature that could combine these various norm representations into a single algorithm. We propose a novel method that integrates these norms into the reinforcement learning process. Our method monitors the agent's compliance with the given norms and summarizes it in a quantity we call the agent's reputation. This quantity is used to weigh the received rewards to motivate the agent to become value-aligned. We carry out a series of experiments including a continuous state space traffic problem to demonstrate the importance of the written and unwritten norms and show how our method can find the value-aligned policies. Furthermore, we carry out ablations to demonstrate why it is better to combine these two groups of norms rather than using either separately.

Figures (10)

• Figure 1: At each time-step Alignment Value $\mathcal{AV}$ receives $s_t, a_t$ from the agent and calculates the new agent's reputation $w_{t+1}$ which then becomes part of the state $s_{t+1}$. An agent's action is sent to the environment only if it is within the permitted rule-based actions. If not, another (closest) rule-based action is executed instead.
• Figure 2: Development of two HAVA agents' reputation $w_t$ in the junction scenario. We can see that $\alpha=0.5$ (15 steps) tends to violate the norms more often (more dips) than $\alpha=0.1$ (45 steps) because it is forgiven faster. This can be seen around the time $t=75$ when both agents violate the $DD$ norms by more than $\tau=1$ and their reputation decreases to 0. The agent with $\alpha=0.5$ recovers its reputation much quicker than the agent with $\alpha=0.1$.
• Figure 3: Toy Example: Grid world with three policies $\pi_R, \pi_Y, \pi_G$. The social norms in $DD$ prohibit visiting the lawn tiles (green). The expected discounted return of the offending policies $\pi_R, \pi_Y$ depends on $\alpha$ while the return of $\pi_G$ remains unchanged.
• Figure 4: The agent is spawned and must learn how to cross the junction in the fastest way while respecting the necessary safety, legal and social norms.
• Figure 5: Simulated human trajectories used to train $DD$ to predict the social norm of speed. The human drivers accelerate (time $t=50$), occasionally violating the speed limit, then stop at the junction to let the north-south vehicle pass (time $t=100$) and finally accelerate to leave the junction (time $t=200$).