Predictive Response Optimization: Using Reinforcement Learning to Fight Online Social Network Abuse

TL;DR

The paper redefines OSN abuse mitigation as action optimization under budgets, not mere detection. It introduces Predictive Response Optimization (PRO), a two-layer system that uses contextual multi-armed bandit action selection at the entity level and model predictive control to enforce global constraints, with Gaussian Process-based reward models to predict outcomes. In real deployments on Instagram and Facebook, PRO substantially reduced abusive scraping (up to 59.2% on Instagram and 4.5% on Facebook) without harming benign users, and demonstrated rapid adaptation to changing business constraints and adversarial tactics. The work provides a scalable, adaptable framework for integrating reinforcement learning into OSN abuse mitigation and offers a pathway for extending these methods to other enforcement problems.

Abstract

Detecting phishing, spam, fake accounts, data scraping, and other malicious activity in online social networks (OSNs) is a problem that has been studied for well over a decade, with a number of important results. Nearly all existing works on abuse detection have as their goal producing the best possible binary classifier; i.e., one that labels unseen examples as "benign" or "malicious" with high precision and recall. However, no prior published work considers what comes next: what does the service actually do after it detects abuse? In this paper, we argue that detection as described in previous work is not the goal of those who are fighting OSN abuse. Rather, we believe the goal to be selecting actions (e.g., ban the user, block the request, show a CAPTCHA, or "collect more evidence") that optimize a tradeoff between harm caused by abuse and impact on benign users. With this framing, we see that enlarging the set of possible actions allows us to move the Pareto frontier in a way that is unattainable by simply tuning the threshold of a binary classifier. To demonstrate the potential of our approach, we present Predictive Response Optimization (PRO), a system based on reinforcement learning that utilizes available contextual information to predict future abuse and user-experience metrics conditioned on each possible action, and select actions that optimize a multi-dimensional tradeoff between abuse/harm and impact on user experience. We deployed versions of PRO targeted at stopping automated activity on Instagram and Facebook. In both cases our experiments showed that PRO outperforms a baseline classification system, reducing abuse volume by 59% and 4.5% (respectively) with no negative impact to users. We also present several case studies that demonstrate how PRO can quickly and automatically adapt to changes in business constraints, system behavior, and/or adversarial tactics.

Figures (3)

• Figure 1: Two conceptual anti-abuse methods tuned under constraints.
• Figure 2: Predictive Response Optimization system design
• Figure 3: (a) Selection rate of the new enforcement action. (b) Daily deltas ($Test-Control$) of the abuse metric automated requests. (c) Cost metric time spent (7-day moving average)