Higher-Order DeepTrails: Unified Approach to *Trails

TL;DR

The paper tackles the limitation of first-order Markov models in capturing higher-order dependencies in sequential user behavior. It introduces a unified framework that trains a small autoregressive language model on observed sequences and uses the frozen model's position-wise cross-entropy loss $L_{\mathrm{CE}}$ to evaluate hypotheses and feature-driven subgroups across three settings: DeepHypTrails, DeepMixedTrails, and DeepSubTrails. Through synthetic datasets and a real-world voice-assistant case study, the approach demonstrates its ability to model higher-order dependencies, perform per-position diagnostics, and handle homogeneous, heterogeneous, and subgroup analyses within a single methodology. The results indicate improved insight into user behavior, enabling better UX design, personalized recommendations, and adaptive interfaces for complex sequential interactions. Overall, the work provides a principled, unified methodology for higher-order sequential analysis with practical impact on real-world interaction data.

Abstract

Analyzing, understanding, and describing human behavior is advantageous in different settings, such as web browsing or traffic navigation. Understanding human behavior naturally helps to improve and optimize the underlying infrastructure or user interfaces. Typically, human navigation is represented by sequences of transitions between states. Previous work suggests to use hypotheses, representing different intuitions about the navigation to analyze these transitions. To mathematically grasp this setting, first-order Markov chains are used to capture the behavior, consequently allowing to apply different kinds of graph comparisons, but comes with the inherent drawback of losing information about higher-order dependencies within the sequences. To this end, we propose to analyze entire sequences using autoregressive language models, as they are traditionally used to model higher-order dependencies in sequences. We show that our approach can be easily adapted to model different settings introduced in previous work, namely HypTrails, MixedTrails and even SubTrails, while at the same time bringing unique advantages: 1. Modeling higher-order dependencies between state transitions, while 2. being able to identify short comings in proposed hypotheses, and 3. naturally introducing a unified approach to model all settings. To show the expressiveness of our approach, we evaluate our approach on different synthetic datasets and conclude with an exemplary analysis of a real-world dataset, examining the behavior of users who interact with voice assistants.

Figures (7)

• Figure 1: DeepTrails: Schematic overview of our approach. We train a small language model on observed user sequences, optionally with user features. Then we freeze the trained model and plug it into the respective setting: 1. DeepHypTrails: evaluating the model on sequences generated from first-order hypothesis, or 2. DeepMixedTrails: using sequences that contain mixed transition behavior, or 3. DeepSubTrails: identifying interesting subgroups of features.
• Figure 2: Applying autoregressive language models, here GPT, to sequential behavior analysis. \ref{['fig:even-biased-decoding']} shows the results trained on sequences with only even behavior and evaluates it on different hypotheses. For each time step, we rank the predicted tokens by logits in descending order. We plot the average rank of the target token per decoding step for each hypothesis. A low rank indicates a high likelihood for the next transition according to the model. \ref{['fig:mixed']} shows the result when training on sequences of heterogeneous behaviors, as introduced in \ref{['sec:exp:mixedtrails']}. The number in the legend indicates the average loss over all sequences and positions for the given hypothesis.
• Figure 3: \ref{['fig:first-even-biased-umap']} shows the ability of our model to learn the graph structure and the different node types. We use a UMAP visualization of node embeddings, which shows a separation between the different node types. \ref{['fig:synheatmap']} shows a visualization of the DeepSubTrails loss scores in synthetic data. The model is trained on sequences prepended with features. For evaluation, we create different combinations of features and walk behavior and visualize the loss. \ref{['fig:amzheatmap']} shows the same visualization in our real-world dataset. Here the x-axis lists all command sequences and the y-axis all distinct feature combinations. \ref{['fig:amzfeatclusters']} clusters the possible feature sets with respect to the column-wise probability given by our model from \ref{['fig:amzheatmap']}.
• Figure 4: \ref{['fig:first-even-biased']} is trained on first-even-biased, which means that the behavior is chosen using 90% and 10% the opposite behavior. \ref{['fig:hyptrails']} shows a HypTrails analysis on higher order sequences. The y-axis shows the evidence (higher is better), and the x-axis the concentration factor. The underlying data were created by the same behaviour as the red hypothesis. We can observe, that the evidence scores for higher order hypothesis (red) and random (purple) are similar. Hence HypTrails is not able to distinguish the higher order dependencies. For a deeper understanding of the plot, we refer to author to DBLP:conf/www/SingerHHS15.
• Figure 5: Visualisation of DeepMixedTrails second setting, where a model is trained on sequences with the same driving force, but containing higher-order dependencies. \ref{['fig:mixed-higher-order']} shows a model trained on first even observations. \ref{['fig:mixed-two-two']} displays the result when training on observations, which transitions twice to even nodes, then two time to an odd node.