Understanding Agent Incentives using Causal Influence Diagrams. Part I: Single Action Settings
Tom Everitt, Pedro A. Ortega, Elizabeth Barnes, Shane Legg
TL;DR
The paper introduces causal influence diagrams (CID) to model how agents' objectives interact with environments and generate incentives. It derives graphical criteria for observation incentives and intervention incentives in single-decision CID graphs, enabling identification of what information should be protected or controlled. The authors prove the 'if and only if' conditions for observation incentives (d-connected to an influenceable utility node given the decision and all available observations) and for intervention incentives (existence of a directed path from X to a utility node after removing nonrequisite links). They illustrate applications to fairness and to preventing a QA system from manipulating the world through its outputs, and discuss how different algorithms shape CID structures. This framework aids in auditing incentives and guiding algorithm design toward safer, more reliable decision-making.
Abstract
Agents are systems that optimize an objective function in an environment. Together, the goal and the environment induce secondary objectives, incentives. Modeling the agent-environment interaction using causal influence diagrams, we can answer two fundamental questions about an agent's incentives directly from the graph: (1) which nodes can the agent have an incentivize to observe, and (2) which nodes can the agent have an incentivize to control? The answers tell us which information and influence points need extra protection. For example, we may want a classifier for job applications to not use the ethnicity of the candidate, and a reinforcement learning agent not to take direct control of its reward mechanism. Different algorithms and training paradigms can lead to different causal influence diagrams, so our method can be used to identify algorithms with problematic incentives and help in designing algorithms with better incentives.