Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Influencing Bandits: Arm Selection for Preference Shaping

Viraj Nadkarni, D. Manjunath, Sharayu Moharir

TL;DR

This work considers a non stationary multi-armed bandit in which the population preferences are positively and negatively reinforced by the observed rewards and describes an Explore-then-commit policy and a Thompson sampling based algorithm for the case when more than two types of opinions are present.

Abstract

We consider a non stationary multi-armed bandit in which the population preferences are positively and negatively reinforced by the observed rewards. The objective of the algorithm is to shape the population preferences to maximize the fraction of the population favouring a predetermined arm. For the case of binary opinions, two types of opinion dynamics are considered -- decreasing elasticity (modeled as a Polya urn with increasing number of balls) and constant elasticity (using the voter model). For the first case, we describe an Explore-then-commit policy and a Thompson sampling policy and analyse the regret for each of these policies. We then show that these algorithms and their analyses carry over to the constant elasticity case. We also describe a Thompson sampling based algorithm for the case when more than two types of opinions are present. Finally, we discuss the case where presence of multiple recommendation systems gives rise to a trade-off between their popularity and opinion shaping objectives.

Influencing Bandits: Arm Selection for Preference Shaping

TL;DR

This work considers a non stationary multi-armed bandit in which the population preferences are positively and negatively reinforced by the observed rewards and describes an Explore-then-commit policy and a Thompson sampling based algorithm for the case when more than two types of opinions are present.

Abstract

We consider a non stationary multi-armed bandit in which the population preferences are positively and negatively reinforced by the observed rewards. The objective of the algorithm is to shape the population preferences to maximize the fraction of the population favouring a predetermined arm. For the case of binary opinions, two types of opinion dynamics are considered -- decreasing elasticity (modeled as a Polya urn with increasing number of balls) and constant elasticity (using the voter model). For the first case, we describe an Explore-then-commit policy and a Thompson sampling policy and analyse the regret for each of these policies. We then show that these algorithms and their analyses carry over to the constant elasticity case. We also describe a Thompson sampling based algorithm for the case when more than two types of opinions are present. Finally, we discuss the case where presence of multiple recommendation systems gives rise to a trade-off between their popularity and opinion shaping objectives.
Paper Structure (27 sections, 12 theorems, 50 equations, 9 figures, 2 algorithms)

This paper contains 27 sections, 12 theorems, 50 equations, 9 figures, 2 algorithms.

Table of Contents

  1. Introduction
  2. Related Work
  3. Model and Preliminaries
  4. Preference Shaping with Known Rewards Matrix
  5. Preference Shaping with Unknown Rewards Matrix
  6. The Explore-Then-Commit (ETC) Algorithm
  7. Thompson Sampling
  8. Simulations
  9. Preference Shaping with Constant Influence
  10. Extensions
  11. Generalizing to $N$ arms
  12. Two competing recommendation systems
  13. Case 1 (Uniform Population)
  14. Case 2 (Polarized Population)
  15. Concluding Remarks
  16. ...and 12 more sections

Key Result

Theorem 1

The optimal policy at time $t$ is where $\mathbbm{1}_{\{\cdot \}}$ is the indicator function.

Figures (9)

  • Figure 1: Trajectory of proportion of type 1 population. Comparison of o.d.e. solution and averages from 1, 10 amd 100 sample paths. We used $B$ with the values $b_{00}=0.9,$$b_{01}=0.4,$$b_{10}=0.2,$ and $b_{11}=0.6.$
  • Figure 2: Expected population proportion vs time (left) and cumulative regret vs time (right) for the ETC, TS, and the optimal policy that knows $B.$ The $B$ used for each of the plots and the optimal policy are also shown.
  • Figure 3: Expected population proportion vs time (left) and cumulative regret vs time (right) for the ETC, TS, and the optimal for $B_{sym} = (b_{11}=0.9, b_{12}=0.7, b_{21}=0.7, b_{22}=0.9)$
  • Figure 4: Expected population proportion vs time for optimal policies that knows $B.$ for DID model and the CID model. The $B$ used for each of the plots is also shown.
  • Figure 5: Expected population proportion vs time for optimal policy and Thompson sampling policy for $N-$arm case. The $N$ and $B$ used for each of the plots is also shown.
  • ...and 4 more figures

Theorems & Definitions (16)

  • Definition 1: Policy
  • Definition 2: Optimal policy
  • Definition 3: One-step regret
  • Definition 4: Cumulative regret
  • Theorem 1
  • Lemma 1
  • Theorem 2
  • Lemma 2
  • Lemma 3
  • Theorem 3
  • ...and 6 more