Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Should Robots be Obedient?

Smitha Milli, Dylan Hadfield-Menell, Anca Dragan, Stuart Russell

TL;DR

The paper formalizes the tradeoff between robot obedience and owner value using a supervision POMDP where robots infer human preferences from orders. It shows that an IRL-based robot can outperform blind obedience when humans are imperfectly rational, and analyzes how learning approaches (MLE, MAP, posterior mean) and model misspecification affect performance and robustness. Key contributions include theoretical guarantees on autonomy advantage, robustness of MLE-based policies to certain misspecifications, and practical strategies (like policy mixing) to detect and adapt to wrong models of human preferences. The work argues for a middle-ground where robots intelligently decide when to obey, prioritizing obedience to remain safe and robust in the face of uncertainty while still pursuing improved outcomes whenever feasible.

Abstract

Intuitively, obedience -- following the order that a human gives -- seems like a good property for a robot to have. But, we humans are not perfect and we may give orders that are not best aligned to our preferences. We show that when a human is not perfectly rational then a robot that tries to infer and act according to the human's underlying preferences can always perform better than a robot that simply follows the human's literal order. Thus, there is a tradeoff between the obedience of a robot and the value it can attain for its owner. We investigate how this tradeoff is impacted by the way the robot infers the human's preferences, showing that some methods err more on the side of obedience than others. We then analyze how performance degrades when the robot has a misspecified model of the features that the human cares about or the level of rationality of the human. Finally, we study how robots can start detecting such model misspecification. Overall, our work suggests that there might be a middle ground in which robots intelligently decide when to obey human orders, but err on the side of obedience.

Should Robots be Obedient?

TL;DR

The paper formalizes the tradeoff between robot obedience and owner value using a supervision POMDP where robots infer human preferences from orders. It shows that an IRL-based robot can outperform blind obedience when humans are imperfectly rational, and analyzes how learning approaches (MLE, MAP, posterior mean) and model misspecification affect performance and robustness. Key contributions include theoretical guarantees on autonomy advantage, robustness of MLE-based policies to certain misspecifications, and practical strategies (like policy mixing) to detect and adapt to wrong models of human preferences. The work argues for a middle-ground where robots intelligently decide when to obey, prioritizing obedience to remain safe and robust in the face of uncertainty while still pursuing improved outcomes whenever feasible.

Abstract

Intuitively, obedience -- following the order that a human gives -- seems like a good property for a robot to have. But, we humans are not perfect and we may give orders that are not best aligned to our preferences. We show that when a human is not perfectly rational then a robot that tries to infer and act according to the human's underlying preferences can always perform better than a robot that simply follows the human's literal order. Thus, there is a tradeoff between the obedience of a robot and the value it can attain for its owner. We investigate how this tradeoff is impacted by the way the robot infers the human's preferences, showing that some methods err more on the side of obedience than others. We then analyze how performance degrades when the robot has a misspecified model of the features that the human cares about or the level of rationality of the human. Finally, we study how robots can start detecting such model misspecification. Overall, our work suggests that there might be a middle ground in which robots intelligently decide when to obey human orders, but err on the side of obedience.

Paper Structure

This paper contains 8 sections, 6 theorems, 14 equations, 5 figures.

Table of Contents

  1. Introduction
  2. Human-Robot Interaction Model
  3. Justifying Autonomy
  4. Approximations via IRL
  5. Model Misspecification
  6. Related Work
  7. Conclusion
  8. Acknowledgements

Key Result

Theorem 1

The optimal robot $\mathbf{R}^{*}{}$ is an IRL-R whose policy $\pi_{\mathbf{R}}^{*}$ has $\hat{\theta}{}$ equal to the posterior mean of $\theta$. $\mathbf{R}^{*}$ is guaranteed a nonnegative advantage on each round: $\forall n$$\Delta_{n} \geq 0$ with equality if and only if $\forall n$$\pi_{\mathb

Figures (5)

  • Figure 1: (Left) The blindly obedient robot always follows $\mathbf{H}$'s order. (Right) An IRL-R computes an estimate of $\mathbf{H}$'s preferences and picks the action optimal for this estimate.
  • Figure 2: Autonomy advantage $\Delta$ (left) and obedience $\mathcal{O}$ (right) over time.
  • Figure 3: When $\mathbf{H}$ is more irrational $\Delta$ converges to a higher value, but at a slower rate.
  • Figure 4: $\Delta$ and $\mathcal{O}$ when $\Theta$ is misspecified
  • Figure 5: (Detecting misspecification) The bold line shows the $\mathbf{R}$ that tries detecting missing features (Equation \ref{['detect-policy']}), as compared to MLE-$\mathbf{R}$ (which is also shown in Figure \ref{['fig:missing_feats']}).

Theorems & Definitions (16)

  • Remark 1
  • Theorem 1
  • proof
  • Theorem 2
  • proof
  • Remark 2
  • Theorem 3
  • proof
  • Lemma 1
  • proof
  • ...and 6 more