Relying on the Metrics of Evaluated Agents

TL;DR

It is shown that an agent will prefer to reveal metrics that differentiate the most difficult tasks from the rest, and conceal metrics that differentiate the easiest, which indicates an economic value to privacy that yields Pareto improvement for both the agent and evaluator.

Abstract

Online platforms and regulators face a continuing problem of designing effective evaluation metrics. While tools for collecting and processing data continue to progress, this has not addressed the problem of "unknown unknowns", or fundamental informational limitations on part of the evaluator. To guide the choice of metrics in the face of this informational problem, we turn to the evaluated agents themselves, who may have more information about how to measure their own outcomes. We model this interaction as an agency game, where we ask: "When does an agent have an incentive to reveal the observability of a metric to their evaluator?" We show that an agent will prefer to reveal metrics that differentiate the most difficult tasks from the rest, and conceal metrics that differentiate the easiest. We further show that the agent can prefer to reveal a metric "garbled" with noise over both fully concealing and fully revealing. This indicates an economic value to privacy that yields Pareto improvement for both the agent and evaluator. We demonstrate these findings on data from online rideshare platforms.

Figures (11)

• Figure 1: Timing of the agency game with information transfer between principal (P) and agent (A).
• Figure 2: Timing of the agency game with garbled information transfer between principal (P) and agent (A).
• Figure 3: Agent's utility differences for revealing $X$ and garbled $Y$ when $C$ is a mixture of exponentials. We fix $\theta=\frac{1}{2}$, $b = 1$, and $\lambda_0 = 0.5$, and vary $\lambda_1$. The solid blue line shows the agents utility difference upon revealing $X$ for each value of $\lambda_1$. The dashed orange line shows the agent's utility difference upon revealing the optimal garbled $Y$, for optimal garbling parameter $\varepsilon^{*}$ displayed above. This first lighter shaded region highlights settings where garbled > concealed > revealed. The second darker shaded region highlights settings where garbled > revealed > concealed .
• Figure 4: Agent's utility differences for revealing the distance feature (positive means the agent prefers to reveal). The dotted line is the agent's value difference upon revealing the full distance feature $X$. The solid blue line shows the agents value differences upon revealing $Z^t$ for different thresholds $t$. The dashed orange line shows the agent's value difference upon revealing the optimal garbled version of $Z^t$, for optimal garbling parameter $\varepsilon^{*}$ displayed above ($\varepsilon = 1$ corresponds to full revelation, and $\varepsilon = 0$ to full concealment). In alignment with our theory, the agent prefers to conceal for $t$ low enough, and reveal for $t$ high enough. There also exist $t$ values in the middle in which garbled > revealed > concealed.
• Figure 5: Difference between agent's utility in the revealed setting and concealed settings when $C \sim \text{Unif}(0,1)$, and $X = X^t$ is a thresholding feature. For a given pair $(b,t)$, a positive utility difference means the agent prefers to reveal $X^t$, and a negative utility difference means the agent prefers to conceal $X^t$. The agent always prefers to conceal for $t$ sufficiently close to $0$, which matches Theorem \ref{['thm:conceal_t']}. For $b \in (1,2)$, the agent prefers to reveal for $t > \frac{b}{2}$, which matches Theorem \ref{['thm:reveal_high_t']}. At $b = 2$, the agent always prefers to conceal. For $b < 1$, the agent actually prefers to conceal for sufficiently high $t$.

Theorems & Definitions (46)

• Theorem 1: Agent prefers to reveal for high thresholds
• Theorem 2: Agent prefers to conceal for low thresholds
• Lemma 1
• Lemma 2: Optimal garbling increases welfare over concealment
• Lemma 3: More information initially increases welfare
• Lemma 4: More information increases total welfare relative to agent optimal garbling
• Definition 1: Local Randomizer kasiviswanathan2011can
• Lemma 5: Privacy of garbling mechanism
• Theorem 3
• proof