BBE-LSWCM: A Bootstrapped Ensemble of Long and Short Window Clickstream Models

TL;DR

The paper tackles real-time SaaS clickstream prediction by unifying long-window historical behavior with in-session activity through a block-bootstrapped ensemble called BBE-LSWCM. It decomposes the problem into a Long Window Model, a Short Window Model, and an Ensemble Meta Model, each producing $p^{l}(T)$, $p^{s}(T)$, and $p^{e}(T)$ respectively, with $p^{e}(T)=\phi^{e}([p^{l}(T), p^{s}(T), z(T)]; \Theta^{e})$, enabling rapid, multi-inference predictions per ref-ts $T$. Key innovations include automatic feature featurization for the long window, BiLSTM-based sequential modeling for the short window, and a profile-likelihood-based parameter estimation regime using block bootstrap sampling to mitigate overfitting while preserving temporal dependencies. On the QBO dataset, BBE-LSWCM achieves substantial gains over baselines in real-time churn detection (DL$1$≈3.20, AUROC≈0.813, ATWC1≈71h) and intended-task detection (AUROC≈0.806 vs 0.701 for SWM), and enables effective in-session proactive interventions that reduce churn by ~30% in online experiments. The approach offers practical impact through a scalable, deployable pipeline that blends streaming real-time inferences with periodically updated historical context to drive context-aware customer interventions.

Abstract

We consider the problem of developing a clickstream modeling framework for real-time customer event prediction problems in SaaS products like QBO. We develop a low-latency, cost-effective, and robust ensemble architecture (BBE-LSWCM), which combines both aggregated user behavior data from a longer historical window (e.g., over the last few weeks) as well as user activities over a short window in recent-past (e.g., in the current session). As compared to other baseline approaches, we demonstrate the superior performance of the proposed method for two important real-time event prediction problems: subscription cancellation and intended task detection for QBO subscribers. Finally, we present details of the live deployment and results from online experiments in QBO.

Figures (4)

• Figure 1: BBE-LSWCM inference mechanism (black rectangles denote ML modules).
• Figure 2: ovr-AUROC of BBE-LSWCM and LWM for varying LWM hyper-parameters
• Figure 3: ovr-AUROC of BBE-LSWCM and SWM for varying SWM hyper-parameters
• Figure 4: BBE-LSWCM production deployment architecture.