Algorithmic Drift: A Simulation Framework to Study the Effects of Recommender Systems on User Preferences

TL;DR

A stochastic simulation framework that mimics user-recommender system interactions in a long-term scenario and formalizes a user model, which comprises behavioral aspects, such as the user resistance towards the recommendation algorithm and their inertia in relying on the received suggestions.

Abstract

Digital platforms such as social media and e-commerce websites adopt Recommender Systems to provide value to the user. However, the social consequences deriving from their adoption are still unclear. Many scholars argue that recommenders may lead to detrimental effects, such as bias-amplification deriving from the feedback loop between algorithmic suggestions and users' choices. Nonetheless, the extent to which recommenders influence changes in users leaning remains uncertain. In this context, it is important to provide a controlled environment for evaluating the recommendation algorithm before deployment. To address this, we propose a stochastic simulation framework that mimics user-recommender system interactions in a long-term scenario. In particular, we simulate the user choices by formalizing a user model, which comprises behavioral aspects, such as the user resistance towards the recommendation algorithm and their inertia in relying on the received suggestions. Additionally, we introduce two novel metrics for quantifying the algorithm's impact on user preferences, specifically in terms of drift over time. We conduct an extensive evaluation on multiple synthetic datasets, aiming at testing the robustness of our framework when considering different scenarios and hyper-parameters setting. The experimental results prove that the proposed methodology is effective in detecting and quantifying the drift over the users preferences by means of the simulation. All the code and data used to perform the experiments are publicly available.

Figures (8)

• Figure 1: Framework overview. Considering the initial interactions of each user $u$, we start the simulation (1) by invoking the trained recsys model B times. In each of these iterations, we get the top-k recommended items (1.1) and we sample an item (1.2), thus updating the history of $u$ (1.3). We repeat the process T times, in order to build a tree whose nodes are the items sampled (1.4), of depth T. Intuitively, each tree represents a pathway of the user $u$ guided by the recsys. After each iteration, we restore the initial user history (1.5), and the process restarts. At the end of the B iterations, we obtain $T$ different trees, representing each pathway (2). Hence, we aggregate all the trees (3) and we obtain the final recommendation graph $G^u$.
• Figure 2: Users' preferences matrix in different samples, varying the proportion of Non-/Semi-/Radicalized users. Assuming that the first (resp. last) $\frac{|I|}{2}$ items are labelled as "neutral" (resp. "harmful"), we impose the Non-Radicalized (resp. Radicalized) users prefer neutral (resp. harmful) items. Conversely, we assume semi-radicalized users span from all items' categories, exhibiting common preferences with the two polarized communities.
• Figure 3: Harmful distribution of Non-/Semi-/Radicalized users, varying the population proportion in the sample. The X-axis represents the harmful percentage in users history, while the Y-axis shows the percentage of users in the dataset.
• Figure 4: Algorithmic Drift Score (ADS) computed over users graphs by varying the proportion of the starting population (Non-/Semi-/Radicalized %).
• Figure 5: Delta Target Consumption (DTC), expressed in percentage, computed by varying the proportion of the starting population (Non-/Semi-/Radicalized %).