AutoLike: Auditing Social Media Recommendations through User Interactions

TL;DR

AutoLike introduces a reinforcement learning framework to audit social media recommendation systems by driving a platform's For You Page toward content that matches a chosen topic and sentiment. By modeling the interaction as an RL problem with state defined as a topic-sentiment pair and a proximity-based reward, AutoLike learns policies that maximize content relevance, producing pathways of content and actions for analysis. The TikTok case study demonstrates automated content classification via Whisper and bart-large-mnli and shows that a streamlined AutoLike can steer recommendations toward targeted topics and negative sentiments, with quantifiable increases over controls. The work provides a practical tool for regulators and platform developers to assess and understand RS behavior, while discussing ethical considerations and future extensions to other platforms and richer state dimensions.

Abstract

Modern social media platforms, such as TikTok, Facebook, and YouTube, rely on recommendation systems to personalize content for users based on user interactions with endless streams of content, such as "For You" pages. However, these complex algorithms can inadvertently deliver problematic content related to self-harm, mental health, and eating disorders. We introduce AutoLike, a framework to audit recommendation systems in social media platforms for topics of interest and their sentiments. To automate the process, we formulate the problem as a reinforcement learning problem. AutoLike drives the recommendation system to serve a particular type of content through interactions (e.g., liking). We apply the AutoLike framework to the TikTok platform as a case study. We evaluate how well AutoLike identifies TikTok content automatically across nine topics of interest; and conduct eight experiments to demonstrate how well it drives TikTok's recommendation system towards particular topics and sentiments. AutoLike has the potential to assist regulators in auditing recommendation systems for problematic content. (Warning: This paper contains qualitative examples that may be viewed as offensive or harmful.)

Figures (6)

• Figure 1: AutoLike Framework. Formulated as a reinforcement learning problem, AutoLike enables auditing of social media recommendation algorithms. Specifically, the user provides a topic of interest and a sentiment, which characterizes the content they want to audit. A RL agent interacts with the environment ( e.g., TikTok's "For You" page). At each time step, the agent follows a RL policy to select which action to apply to the current recommended content, then swipes to the next content. It receives a reward that reflects whether the algorithm recommends content related to the given inputs ( e.g., high rewards for on-topic content). It does this for a specified time horizon and learns over time which actions most efficiently drive the algorithm. The output of AutoLike are pathways: sequences of recommended content and the actions taken upon them. The user can further analyze the pathways to understand the RS.
• Figure 2: (a) We provide a conceptual example of how AutoLike can drive the RS to an end goal (red box) across two dimensions: a topic of interest (eating disorder) and sentiment (negative/sad), using real TikTok content. The RS starts at a benign state for a new user, serving popular common content, such as cats. It skips this content until it reaches more on-topic content, such as ones about dieting. During this time, it begins to like the content, and over time, drives the RS to serve content related to eating disorder. With this example, the negative sentiment comes from the content creator discussing the hardship of recovery without support from family and friends. (b) We illustrate three potential pathways of using AutoLike: (1) $P_{topic}$ shows how AutoLike can drive the RS to be on-topic without considering sentiment; (2) $P_{sentiment}$ shows driving the RS towards negative sentiment content across different topics that are not on-topic; and (3) $P_{both}$ shows the intuitive use case of AutoLike, driving it towards both on-topic and negative sentiment for auditing. $P_{both}$ matches the example within (a).
• Figure 3: AutoLike Implementation. We implement AutoLike for the TikTok Android app. It works in the following ways. Once the user gives the inputs, AutoLike (1) opens TikTok to its "For You" page and extracts the TikTok URL of the current recommended content; (2) sends it to a backend server and the server visits the TikTok URL in a web browser; (3) extracts the text description and transcribes the video's audio into text, and concatenates both into one string; (5) classifies the text into $\langle$Topic, Sentiment$\rangle$; and (6) calculates the reward, and follows a policy to select and take an action ( e.g., liking the current TikTok); and (7) scrolls to the next TikTok. AutoLike repeats 1--7 until some time horizon $T$. We omit the output of pathways for brevity, see Fig. \ref{['fig:autolike-framework']}.
• Figure 4: Zero-shot Classification. We treat the x-axis as the topic of interest (9) and the #hashtag as the ground truth. The y-axis is the classification confidence score that a TikTok is on-topic or related to the topic of interest. For example, for the left-most subplot, we expect when "mental health" is given as the topic of interest, that the majority of TikToks will have higher confidence scores since the TikToks were collected using the hashtag #mentalhealth. Expected results: "discrimination" should have higher confidence scores classifying TikToks from #hatespeech, "eating disorder" for #ed, and all topics of interest are low confidence for #foryou. Overall, we evaluated 24 different hashtags.
• Figure 5: AutoLike for Single Dimensions: TikTok. We demonstrate how AutoLike can drive TikTok's RS across individual dimensions, either for topic of interest only or for sentiment only. The y-axis denotes the number of times we liked the TikTok video ( i.e., ones we classify as on-topic or on-sentiment with confidence scores $> 0.5$). We provide thumbnail previews of TikToks at Fig. \ref{['fig:pets-thumbnail']} and Fig. \ref{['fig:sad-thumbnail']}.