Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

General-Purpose User Modeling with Behavioral Logs: A Snapchat Case Study

Qixiang Fang, Zhihan Zhou, Francesco Barbieri, Yozen Liu, Leonardo Neves, Dong Nguyen, Daniel L. Oberski, Maarten W. Bos, Ron Dotsch

TL;DR

This paper investigates applying general-purpose BLUM (G-BLUM) to instant messaging platforms using Snapchat behavior logs to learn user representations. It introduces a Transformer-based upstream model trained with Masked Behavior Prediction and User Contrastive Learning, and uses Attention with Linear Biases (ALiBi) to enable efficient extrapolation to longer sequences. Three novel downstream tasks—Reported Account Prediction, Ad View Time Prediction, and Account Self-deletion Prediction—probe user safety, engagement, and churn. Across six evaluation datasets, the approach yields high-quality representations and shows that sequences as short as 128 events suffice, with ALiBi enhancing long-sequence inference and ablating effects confirming the value of UCL and ALiBi.

Abstract

Learning general-purpose user representations based on user behavioral logs is an increasingly popular user modeling approach. It benefits from easily available, privacy-friendly yet expressive data, and does not require extensive re-tuning of the upstream user model for different downstream tasks. While this approach has shown promise in search engines and e-commerce applications, its fit for instant messaging platforms, a cornerstone of modern digital communication, remains largely uncharted. We explore this research gap using Snapchat data as a case study. Specifically, we implement a Transformer-based user model with customized training objectives and show that the model can produce high-quality user representations across a broad range of evaluation tasks, among which we introduce three new downstream tasks that concern pivotal topics in user research: user safety, engagement and churn. We also tackle the challenge of efficient extrapolation of long sequences at inference time, by applying a novel positional encoding method.

General-Purpose User Modeling with Behavioral Logs: A Snapchat Case Study

TL;DR

This paper investigates applying general-purpose BLUM (G-BLUM) to instant messaging platforms using Snapchat behavior logs to learn user representations. It introduces a Transformer-based upstream model trained with Masked Behavior Prediction and User Contrastive Learning, and uses Attention with Linear Biases (ALiBi) to enable efficient extrapolation to longer sequences. Three novel downstream tasks—Reported Account Prediction, Ad View Time Prediction, and Account Self-deletion Prediction—probe user safety, engagement, and churn. Across six evaluation datasets, the approach yields high-quality representations and shows that sequences as short as 128 events suffice, with ALiBi enhancing long-sequence inference and ablating effects confirming the value of UCL and ALiBi.

Abstract

Learning general-purpose user representations based on user behavioral logs is an increasingly popular user modeling approach. It benefits from easily available, privacy-friendly yet expressive data, and does not require extensive re-tuning of the upstream user model for different downstream tasks. While this approach has shown promise in search engines and e-commerce applications, its fit for instant messaging platforms, a cornerstone of modern digital communication, remains largely uncharted. We explore this research gap using Snapchat data as a case study. Specifically, we implement a Transformer-based user model with customized training objectives and show that the model can produce high-quality user representations across a broad range of evaluation tasks, among which we introduce three new downstream tasks that concern pivotal topics in user research: user safety, engagement and churn. We also tackle the challenge of efficient extrapolation of long sequences at inference time, by applying a novel positional encoding method.
Paper Structure (25 sections, 1 figure, 3 tables)

This paper contains 25 sections, 1 figure, 3 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Conceptualizing G-BLUM Systems
  4. Experiments
  5. Snapchat and Data
  6. Training Dataset
  7. Evaluation Datasets
  8. Transformer-based User Modeling
  9. Training Objectives
  10. ALiBi
  11. Ablation Analysis
  12. Baselines and User Representation Extraction Methods
  13. User Representation Extraction
  14. Model Evaluation
  15. Masked Behavior Prediction
  16. ...and 10 more sections

Figures (1)

  • Figure 1: Model Performance on (a) User Retrieval across Input Sequences Lengths and (b) Three Downstream Tasks across time gaps.