Envious Explore and Exploit

TL;DR

This work analyzes envy in explore-and-exploit multi-agent bandit-like systems with reward consistency, introducing arrival-order mechanisms (uniform, nudged, adversarial) and studying envy as the maximal cumulative reward gap under anonymous algorithms. It derives tight upper and lower bounds for envy under uniform arrival using martingale- and excursion-based analysis, shows that nudging can bound envy independently of the horizon, and proves linear envy growth under adversarial arrival. The paper also initiates a welfare-envy tradeoff, presenting a two-agent optimal welfare scheme and envy-bounded variants that achieve notable welfare gains while constraining envy, with empirical validation through simulations. The results highlight practical implications for fairness in recommendation systems and offer a foundation for extensions to more complex, realistic settings, including MDP-like reward dynamics and varied arrival patterns.

Abstract

Explore-and-exploit tradeoffs play a key role in recommendation systems (RSs), aiming at serving users better by learning from previous interactions. Despite their commercial success, the societal effects of explore-and-exploit mechanisms are not well understood, especially regarding the utility discrepancy they generate between different users. In this work, we measure such discrepancy using the economic notion of envy. We present a multi-armed bandit-like model in which every round consists of several sessions, and rewards are realized once per round. We call the latter property reward consistency, and show that the RS can leverage this property for better societal outcomes. On the downside, doing so also generates envy, as late-to-arrive users enjoy the information gathered by early-to-arrive users. We examine the generated envy under several arrival order mechanisms and virtually any anonymous algorithm, i.e., any algorithm that treats all similar users similarly without leveraging their identities. We provide tight envy bounds on uniform arrival and upper bound the envy for nudged arrival, in which the RS can affect the order of arrival by nudging its users. Furthermore, we study the efficiency-fairness trade-off by devising an algorithm that allows constant envy and approximates the optimal welfare in restricted settings. Finally, we validate our theoretical results empirically using simulations.

Figures (5)

• Figure 1: $\mathcal{E}^t$ as a function of $t$ for both the uniform instance ($I_U$, left panel) and the Bernoulli instance ($I_B$, right panel), each with $N=2$. The three arrival functions shown are $\mathcal{O}_{\textnormal{Uni}}$, $\mathcal{O}_{\textnormal{Ndg}}$ (with $\delta=\frac{1}{2}$), and $\mathcal{O}_{\textnormal{Adv}}$. Green 'X' markers represent the maximum likelihood estimates (MLE) for the linear model $y = c \cdot x$, while orange circles indicate the MLE for the square-root model $y = c \cdot \sqrt{x}$. The perfect alignment of the simulated data with these curves confirms our theoretical predictions.
• Figure 2: $\mathcal{E}^T$ as a function of $N$ for both $I_U$ and $I_B$, under $\mathcal{O}_{\textnormal{Uni}}$ (left panel) and $\mathcal{O}_{\textnormal{Ndg}}$ with $\delta=1/2$ (right panel).
• Figure 3: Sensitivity Analysis for $\mathcal{O}_{\textnormal{Ndg}}$.
• Figure 4: $R_{1}^t+R_{2}^t$ under the $EFC$ algorithm with $C=1$, as a function of $t$.
• Figure 5: $\frac{R_{1}^t+R_{2}^t}{t}$ under the $EFC$ algorithm, as a function of $t$ compared to $1 + \frac{1}{8}\cdot \frac{2C-1}{2C}$. and $1+\frac{1}{8}$

Theorems & Definitions (65)

• Example 1
• Remark 1
• Theorem 1
• proof : Proof of Theorem \ref{['thm: uni upper-bound']}
• Proposition 1
• Claim 1
• Definition 1: Explore-first
• Proposition 2
• Corollary 1
• Definition 2: Sufficiently Random