Aligning AI Agents via Information-Directed Sampling

TL;DR

This work defines a class of bandit alignment problems as an extension of classic multi-armed bandit problems and studies trade-offs theoretically and empirically in a toy bandit alignment problem which resembles the beta-Bernoulli bandit.

Abstract

The staggering feats of AI systems have brought to attention the topic of AI Alignment: aligning a "superintelligent" AI agent's actions with humanity's interests. Many existing frameworks/algorithms in alignment study the problem on a myopic horizon or study learning from human feedback in isolation, relying on the contrived assumption that the agent has already perfectly identified the environment. As a starting point to address these limitations, we define a class of bandit alignment problems as an extension of classic multi-armed bandit problems. A bandit alignment problem involves an agent tasked with maximizing long-run expected reward by interacting with an environment and a human, both involving details/preferences initially unknown to the agent. The reward of actions in the environment depends on both observed outcomes and human preferences. Furthermore, costs are associated with querying the human to learn preferences. Therefore, an effective agent ought to intelligently trade-off exploration (of the environment and human) and exploitation. We study these trade-offs theoretically and empirically in a toy bandit alignment problem which resembles the beta-Bernoulli bandit. We demonstrate while naive exploration algorithms which reflect current practices and even touted algorithms such as Thompson sampling both fail to provide acceptable solutions to this problem, information-directed sampling achieves favorable regret.

Figures (4)

• Figure 1: The above diagram depicts the beta-Bernoulli bandit alignment problem. Each bandit arm has a probability $\phi_i \in [0,1]$ such that $\mathbb{P}(O_{t+1}=1|\phi_i, A_t\in\mathcal{A}_e) = \phi_i$. Meanwhile, the human exhibits a preference $\theta_i \in [0,1]$ for each of the arms which impacts the mean (unobserved) reward of said arm. For all $t$, the agent selects an action $A_t$ which either interfaces with the environment or the human and receives a corresponding observation $O_{t+1}$. The reward is $-1$ if the agent queries the human and unobserved (but dependent on $O_{t+1}, \theta_1$) if the agent interacts with the environment.
• Figure 2: The above plot depicts the performance of IDS and the "explore then exploit" agent for $2$ different values of $\tau$. Evidently, the regret accumulated in the reward-blind exploration phase is very costly. Furthermore, figure \ref{['fig:exploreExploit']} demonstrates that even with this exploration, the "explore then exploit" agents were unable to identify the best action. The shaded area denotes $\pm$ standard error.
• Figure 3: The above plot depicts the cumulative regret of "explore then exploit" agents for $\tau = 3200, 16000$. Note that both axes are in logarithmic scale and hence the dotted line depicts $2t$, exemplified by the slope of $1$. We notice that in the exploration phase, the regret is linear (since it is also slope $1$) and in the same insues in the exploitation phase (slope increase up to $1$ again as $t$ increases).
• Figure 4: The above plot depicts the cumulative regret of IDS. Note that both axes are in logarithmic scale and hence the dotted line depicts $16\sqrt{t}$, exemplified by the slope of $1/2$. The plot suggests that the cumulative regret of IDS is $O(\sqrt{t})$.

Theorems & Definitions (34)

• Lemma 1
• Lemma 2
• Theorem 3
• Lemma 4
• Theorem 5
• Theorem 6
• Lemma 7
• Lemma 8
• Theorem 9
• Theorem 9