Measuring Strategization in Recommendation: Users Adapt Their Behavior to Shape Future Content

TL;DR

The paper investigates whether users strategically adapt their engagement with content to influence future recommendations, challenging the assumption that engagement reflects content quality alone. It combines a controlled lab study using a custom music player with a post-hoc survey to test how information about the learning algorithm and incentives to affect downstream outcomes shape behavior, modelling users as either naive or strategic over a horizon of $T$ interactions. The authors formalize a forward-looking framework, derive two testable hypotheses related to information exposure and horizon length, and provide robust evidence that users strategically adjust engagement (likes, dislikes, dwell time, and related metrics) in response to both information and incentive manipulations. The findings have broad managerial implications, highlighting that algorithm-aware behavior can distort standard engagement signals and that platforms should account for strategization when interpreting user data and designing learning objectives.

Abstract

Most modern recommendation algorithms are data-driven: they generate personalized recommendations by observing users' past behaviors. A common assumption in recommendation is that how a user interacts with a piece of content (e.g., whether they choose to "like" it) is a reflection of the content, but not of the algorithm that generated it. Although this assumption is convenient, it fails to capture user strategization: that users may attempt to shape their future recommendations by adapting their behavior to the recommendation algorithm. In this work, we test for user strategization by conducting a lab experiment and survey. To capture strategization, we adopt a model in which strategic users select their engagement behavior based not only on the content, but also on how their behavior affects downstream recommendations. Using a custom music player that we built, we study how users respond to different information about their recommendation algorithm as well as to different incentives about how their actions affect downstream outcomes. We find strong evidence of strategization across outcome metrics, including participants' dwell time and use of "likes." For example, participants who are told that the algorithm mainly pays attention to "likes" and "dislikes" use those functions 1.9x more than participants told that the algorithm mainly pays attention to dwell time. A close analysis of participant behavior (e.g., in response to our incentive conditions) rules out experimenter demand as the main driver of these trends. Further, in our post-experiment survey, nearly half of participants self-report strategizing "in the wild," with some stating that they ignore content they actually like to avoid over-recommendation of that content in the future. Together, our findings suggest that user strategization is common and that platforms cannot ignore the effect of their algorithms on user behavior.

Figures (13)

• Figure 1: The music player interface with which participants interact.
• Figure 2: (a) The study description that participants in the "Treatment" Incentive condition see. These participants are told that their behaviors will be used to generate personalized music recommendations at the end of the study. (b) The study description that participants in the "Control" Incentive condition see. These participants are told that their behaviors are used to learn what music the general population likes. The participants are randomly divided into the "Treatment"and "Control" Incentive conditions.
• Figure 3: Participants undergo two listening sessions. The first session for all participants is the Warm-up session, as shown in (a). In the second session, participants are randomly assigned to one of three Information conditions, as shown in (b), (c), and (d). (The descriptions above are shown to participants in the "Treatment" Incentive condition. The "Control" Incentive descriptions are analogous.)
• Figure 4: Means and 95% confidence intervals (CIs) across our conditions for our five outcome variables of interest, as described in Section \ref{['sec:outcome_vars']} and using the models in Section \ref{['sec:methods-ate']}.
• Figure 5: Effect of the "Likes" Information condition and the "Dwell" Information condition, compared to the "Control" Information condition, on participant behavior. Left: Average marginal effects of Information conditions on dwell time outcomes. Models are estimated using an OLS regression with controls for behavior in the Warm-up session. Right: Average marginal effects of Information conditions on engagement outcomes. Models are estimated using a quasi-poisson regression with controls for behavior in the Warm-up session.