Machine Learning to Predict Digital Frustration from Clickstream Data

TL;DR

This work tackles the detection of digital frustration from clickstream data, addressing the need for early and accurate predictions to improve user experience. It defines frustration with rule-based signals, transforms raw events into sessions, and engineers a rich set of features including n-grams, HVG motifs, and cyclical time features, then compares classical models (Logistic Regression, Random Forest, XGBoost) with a discriminative LSTM. XGBoost achieves near-baseline performance (ROC AUC ~0.958) while the LSTM attains the best overall accuracy (~0.971 AUC), with substantial gains observed when predicting within the first 20–30 interactions. The results demonstrate that reliable frustration prediction is possible early in a session, enabling real-time interventions and offering a reproducible pipeline using a large Coveo e-commerce clickstream dataset.

Abstract

Many businesses depend on their mobile apps and websites, so user frustration while trying to complete a task on these channels can cause lost sales and complaints. In this research, I use clickstream data from a real e-commerce site to predict whether a session is frustrated or not. Frustration is defined using certain rules based on rage bursts, back and forth navigation (U turns), cart churn, search struggle, and long wandering sessions, and applies these rules to 5.4 million raw clickstream events (304,881 sessions). From each session, I build tabular features and train standard classifier models. I also use the full event sequence to train a discriminative LSTM classifier. XGBoost reaches about 90% accuracy, ROC AUC of 0.9579, while the LSTM performs best with about 91% accuracy and a ROC AUC of 0.9705. Finally, the research shows that with only the first 20 to 30 interactions, the LSTM already predicts frustration reliably.

Figures (5)

• Figure 1: The training vs validation curve shows that the XGBoost model trains very well
• Figure 2: Permutation feature importance and gain importance on XGBoost shows that the features P(view), P(view to detail), hz, z2, z3 are important for model performance, say accuracy, and the features hz, z2, z4, P(add to add) reduce the loss the most.
• Figure 3: The SHAP summary plot shows the features that matter the most overall. Lower values of P(view) influence class 1 (frustration) and higher values influence class 0 (non-frustration). Similarly, from top to bottom, we can see the feature our XGBoost model relies on the most and its influence on both classes.
• Figure 4: Training vs validation loss curve for the LSTM Classifier shows that it trains and generalizes very well.
• Figure 5: The combined ROC curve shows that both XGBoost and LSTM separate the classes (0, 1) very well, with curves close to the top-left corner. The LSTM curve is slightly above the XGBoost curve (AUC 0.97 vs 0.96), meaning that for the same false positive rate, LSTM usually catches a bit more frustrated sessions, so it is the slightly better classifier overall.