Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

USE: Dynamic User Modeling with Stateful Sequence Models

Zhihan Zhou, Qixiang Fang, Leonardo Neves, Francesco Barbieri, Yozen Liu, Han Liu, Maarten W. Bos, Ron Dotsch

TL;DR

User Stateful Embedding (USE) generates user embeddings and reflects users' evolving behaviors without the need for exhaustive reprocessing by storing previous model states and revisiting them in the future, and introduces a novel training objective named future W-behavior prediction to transcend the limitations of next-token prediction.

Abstract

User embeddings play a crucial role in user engagement forecasting and personalized services. Recent advances in sequence modeling have sparked interest in learning user embeddings from behavioral data. Yet behavior-based user embedding learning faces the unique challenge of dynamic user modeling. As users continuously interact with the apps, user embeddings should be periodically updated to account for users' recent and long-term behavior patterns. Existing methods highly rely on stateless sequence models that lack memory of historical behavior. They have to either discard historical data and use only the most recent data or reprocess the old and new data jointly. Both cases incur substantial computational overhead. To address this limitation, we introduce User Stateful Embedding (USE). USE generates user embeddings and reflects users' evolving behaviors without the need for exhaustive reprocessing by storing previous model states and revisiting them in the future. Furthermore, we introduce a novel training objective named future W-behavior prediction to transcend the limitations of next-token prediction by forecasting a broader horizon of upcoming user behaviors. By combining it with the Same User Prediction, a contrastive learning-based objective that predicts whether different segments of behavior sequences belong to the same user, we further improve the embeddings' distinctiveness and representativeness. We conducted experiments on 8 downstream tasks using Snapchat users' behavioral logs in both static (i.e., fixed user behavior sequences) and dynamic (i.e., periodically updated user behavior sequences) settings. We demonstrate USE's superior performance over established baselines. The results underscore USE's effectiveness and efficiency in integrating historical and recent user behavior sequences into user embeddings in dynamic user modeling.

USE: Dynamic User Modeling with Stateful Sequence Models

TL;DR

User Stateful Embedding (USE) generates user embeddings and reflects users' evolving behaviors without the need for exhaustive reprocessing by storing previous model states and revisiting them in the future, and introduces a novel training objective named future W-behavior prediction to transcend the limitations of next-token prediction.

Abstract

User embeddings play a crucial role in user engagement forecasting and personalized services. Recent advances in sequence modeling have sparked interest in learning user embeddings from behavioral data. Yet behavior-based user embedding learning faces the unique challenge of dynamic user modeling. As users continuously interact with the apps, user embeddings should be periodically updated to account for users' recent and long-term behavior patterns. Existing methods highly rely on stateless sequence models that lack memory of historical behavior. They have to either discard historical data and use only the most recent data or reprocess the old and new data jointly. Both cases incur substantial computational overhead. To address this limitation, we introduce User Stateful Embedding (USE). USE generates user embeddings and reflects users' evolving behaviors without the need for exhaustive reprocessing by storing previous model states and revisiting them in the future. Furthermore, we introduce a novel training objective named future W-behavior prediction to transcend the limitations of next-token prediction by forecasting a broader horizon of upcoming user behaviors. By combining it with the Same User Prediction, a contrastive learning-based objective that predicts whether different segments of behavior sequences belong to the same user, we further improve the embeddings' distinctiveness and representativeness. We conducted experiments on 8 downstream tasks using Snapchat users' behavioral logs in both static (i.e., fixed user behavior sequences) and dynamic (i.e., periodically updated user behavior sequences) settings. We demonstrate USE's superior performance over established baselines. The results underscore USE's effectiveness and efficiency in integrating historical and recent user behavior sequences into user embeddings in dynamic user modeling.
Paper Structure (34 sections, 5 equations, 3 figures, 8 tables)

This paper contains 34 sections, 5 equations, 3 figures, 8 tables.

Table of Contents

  1. Introduction
  2. Preliminaries
  3. Behavior-based User Modeling
  4. Stateless and Stateful User Modeling
  5. Method
  6. Model Architecture
  7. Training Objectives
  8. Future $W$-Behavior Prediction
  9. Classification or Regression:
  10. Behaviors of Interest:
  11. Same User Prediction
  12. Implementation
  13. Experiments
  14. Data
  15. Baselines
  16. ...and 19 more sections

Figures (3)

  • Figure 1: Illustration of behavior-based user embedding. We aim to represent each user with a fixed-size numerical vector based on their behavior sequences.
  • Figure 2: Visualization of dynamic user modeling. The empty rectangles indicate user actions; the round shapes indicate user embedding; the drum shapes indicate user states.
  • Figure 3: Accumulated time costs of generating user embeddings in dynamic settings, across USE and three stateless strategies.