PP3D: An In-Browser Vision-Based Defense Against Web Behavior Manipulation Attacks

TL;DR

This work tackles the under-studied challenge of web-based behavior-manipulation attacks (BMAs) by introducing Pixel Patrol 3D (PP3D), a three-component, in-browser defense framework consisting of data discovery (PP_dis), multimodal detection (PP_det), and real-time defense (PP_def). The detection component merges visual cues from screenshots with OCR-extracted text in a compact, resolution-agnostic fusion network suitable for on-device inference, achieving high accuracy and strong generalization to unseen campaigns and screen sizes. The system is trained on a large, labeled BMA dataset and deployed as a privacy-preserving browser extension, with extensive evaluation showing near-perfect AUROC, robust temporal and campaign generalization, and acceptable latency and resource usage across devices. Adversarial robustness is explored, with adversarial training improving resilience to coordinated image and text perturbations. PP3D demonstrates a practical, scalable, and privacy-preserving defense against a broad class of BMAs, and the authors plan to open-source code and datasets to foster further research.

Abstract

Web-based behavior-manipulation attacks (BMAs) - such as scareware, fake software downloads, tech support scams, etc. - are a class of social engineering (SE) attacks that exploit human decision-making vulnerabilities. These attacks remain under-studied compared to other attacks such as information harvesting attacks (e.g., phishing) or malware infections. Prior technical work has primarily focused on measuring BMAs, offering little in the way of generic defenses. To address this gap, we introduce Pixel Patrol 3D (PP3D), the first end-to-end browser framework for discovering, detecting, and defending against behavior-manipulating SE attacks in real time. PP3D consists of a visual detection model implemented within a browser extension, which deploys the model client-side to protect users across desktop and mobile devices while preserving privacy. Our evaluation shows that PP3D can achieve above 99% detection rate at 1% false positives, while maintaining good latency and overhead performance across devices. Even when faced with new BMA samples collected months after training the detection model, our defense system can still achieve above 97% detection rate at 1% false positives. These results demonstrate that our framework offers a practical, effective, and generalizable defense against a broad and evolving class of web behavior-manipulation attacks.

Figures (5)

• Figure 1: Overview of the Pixel Patrol 3D Framework. The $\text{PP}_{det}$ module includes both preprocessing steps (e.g., image normalization and OCR) and the core detection model. The colored blocks within $\text{PP}_{det}$ represent the core model components used for feature extraction and classification.
• Figure 2: Examples of screenshots from each of the 10 randomly selected test campaigns.
• Figure 3: CDF of total inference latency for $\text{PP}_{def}$. Dashed lines indicate median (P50) and 95th-percentile (P95) thresholds.
• Figure 4: Difference in resource usage compared to the baseline during testing. CPU usage is measured as a percentage of total logical cores (e.g., 800% = full use of 8 cores).
• Figure 5: Adversarial examples generated using PGD at increasing perturbation strengths ($\epsilon$).