Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

A Unified Causal Framework for Auditing Recommender Systems for Ethical Concerns

Vibhhu Sharma, Shantanu Gupta, Nil-Jana Akpinar, Zachary C. Lipton, Liu Leqi

TL;DR

This paper views recommender system auditing from a causal lens and provides a general recipe for defining auditing metrics, and proposes two classes of such metrics: future- and past-reacheability and stability, that measure the ability of a user to influence their own and other users' recommendations, respectively.

Abstract

As recommender systems become widely deployed in different domains, they increasingly influence their users' beliefs and preferences. Auditing recommender systems is crucial as it not only ensures the continuous improvement of recommendation algorithms but also safeguards against potential issues like biases and ethical concerns. In this paper, we view recommender system auditing from a causal lens and provide a general recipe for defining auditing metrics. Under this general causal auditing framework, we categorize existing auditing metrics and identify gaps in them -- notably, the lack of metrics for auditing user agency while accounting for the multi-step dynamics of the recommendation process. We leverage our framework and propose two classes of such metrics:future- and past-reacheability and stability, that measure the ability of a user to influence their own and other users' recommendations, respectively. We provide both a gradient-based and a black-box approach for computing these metrics, allowing the auditor to compute them under different levels of access to the recommender system. In our experiments, we demonstrate the efficacy of methods for computing the proposed metrics and inspect the design of recommender systems through these proposed metrics.

A Unified Causal Framework for Auditing Recommender Systems for Ethical Concerns

TL;DR

This paper views recommender system auditing from a causal lens and provides a general recipe for defining auditing metrics, and proposes two classes of such metrics: future- and past-reacheability and stability, that measure the ability of a user to influence their own and other users' recommendations, respectively.

Abstract

As recommender systems become widely deployed in different domains, they increasingly influence their users' beliefs and preferences. Auditing recommender systems is crucial as it not only ensures the continuous improvement of recommendation algorithms but also safeguards against potential issues like biases and ethical concerns. In this paper, we view recommender system auditing from a causal lens and provide a general recipe for defining auditing metrics. Under this general causal auditing framework, we categorize existing auditing metrics and identify gaps in them -- notably, the lack of metrics for auditing user agency while accounting for the multi-step dynamics of the recommendation process. We leverage our framework and propose two classes of such metrics:future- and past-reacheability and stability, that measure the ability of a user to influence their own and other users' recommendations, respectively. We provide both a gradient-based and a black-box approach for computing these metrics, allowing the auditor to compute them under different levels of access to the recommender system. In our experiments, we demonstrate the efficacy of methods for computing the proposed metrics and inspect the design of recommender systems through these proposed metrics.
Paper Structure (47 sections, 3 theorems, 42 equations, 16 figures, 3 tables, 4 algorithms)

This paper contains 47 sections, 3 theorems, 42 equations, 16 figures, 3 tables, 4 algorithms.

Table of Contents

  1. Introduction
  2. Related Work
  3. Structural Causal Models (SCMs).
  4. Recommender System Evaluation.
  5. Auditing Recommender Systems.
  6. A Causal Perspective on Auditing Recommender Systems
  7. Recommender System Setup
  8. Causal Perspective on Auditing Recommender Systems
  9. User-Centric Causal MetricsUser-Centric Causal Metrics for Auditing User Agency
  10. Metrics on User Agency:Past- and Future-Reachability
  11. Metrics on User Influence:Past- and Future-(In)stability
  12. Operationalized Procedure for Computing User-CentricUser Agency Metrics
  13. White-box Access
  14. Black-box Access
  15. Experiments
  16. ...and 32 more sections

Key Result

Proposition 5.1

When $\{\mathbf{p}_i\}_{i \in [n]}$ and $\{\mathbf{q}_j\}_{j \in \mathcal{V}}$ are learned through matrix factorization, the past-$k$ reachability objective eq:past-reachability is concave in the parameters of $f_{1:k}$ if item embeddings $\{\mathbf{q}_j\}_{j \in \mathcal{V}}$ are fixed and $\mathbf

Figures (16)

  • Figure 1: The figure above depicts the causal graph representing a general recommender system at time $t$, specifically pertaining to a user $i$'s interactions with the system. Here, $D_{-i,t}$ denotes the interaction history of all users besides $i$ upto $t$, $A_{i,t}$ denotes the set of recommendations $i$ receives at $t$, $H_{i,t}$ denotes $i$'s interaction history upto $t$, $O_{i,t}$ denotes $i$'s feedback at this time and $o_{i}^{*}$ denotes $i$'s true preference.
  • Figure 2: Scatterplot of Past-5-Reachability for a Matrix Factorization based recommender as $\beta$ varies
  • Figure 3: \ref{['fig:reach_mezo']}, \ref{['fig:stab_mezo']} show how black box access compares to gradient descent (GD) when attempting to estimate past reachability (\ref{['fig:reach_mezo']}) or stability (\ref{['fig:stab_mezo']})
  • Figure 4: Here, we compare the values of Future Reachability and Past Instability for two different values of time horizon and show how reachability and instability increase with longer time horizons.
  • Figure 5: Effect of varying stochasticity on past-stability for a MF recommender system.
  • ...and 11 more figures

Theorems & Definitions (10)

  • Example 1
  • Definition 4.1: future-$k$ reachability
  • Definition 4.2: Past-$k$ reachability
  • Definition 4.3: Future $k$-(In)stability
  • Definition 4.4: Past $k$-(In)stability
  • Proposition 5.1
  • Proposition 5.2
  • proof : Proof of Proposition \ref{['prop:past-recheability']}
  • Lemma A.1
  • proof : Proof of Proposition \ref{['prop:past-stability']}