Lattice: A Confidence-Gated Hybrid System for Uncertainty-Aware Sequential Prediction with Behavioral Archetypes

TL;DR

Lattice tackles uncertainty in sequential prediction by pairing an LSTM with behavioral archetypes and gating their activation via a percentile-based confidence measure. The binary confidence gate ensures archetypes are invoked only when embeddings closely match trained archetypes, otherwise deferring to the baseline. Across MovieLens, LIGO, finance, and cross-domain Amazon Reviews, Lattice shows strong gains when structure is missing (notably with LSTM backbones) and safe deferral or no degradation when structure is already captured by transformers or under distribution shift. This bidirectional validation demonstrates confidence gating as a practical architectural principle for trustworthy, uncertainty-aware sequential models in safety-critical and non-stationary environments.

Abstract

We introduce Lattice, a hybrid sequential prediction system that conditionally activates learned behavioral structure using binary confidence gating. The system clusters behavior windows into behavioral archetypes and uses binary confidence gating to activate archetype-based scoring only when confidence exceeds a threshold, falling back to baseline predictions when uncertain. We validate Lattice on recommendation systems (MovieLens), scientific time-series (LIGO), and financial markets, using LSTM and transformer backbones. On MovieLens with LSTM, Lattice achieves +31.9% improvement over LSTM baseline in HR@10 (p < 3.29 x 10^-25, 30 seeds), outperforming transformer baselines by 109.4% over SASRec and 218.6% over BERT4Rec. On LIGO and financial data, the system correctly refuses archetype activation when distribution shift occurs - a successful outcome demonstrating confidence gating prevents false activation. On transformer backbones, Lattice provides 0.0% improvement (neutral, no degradation), gracefully deferring when structure is already present. This bidirectional validation - activating when patterns apply, refusing when they don't, and deferring when redundant - supports confidence gating as a promising architectural principle for managing epistemic uncertainty in safety-critical applications.

Figures (1)

• Figure 1: Lattice data flow: confidence-gated archetype activation. Given a sequence, the LSTM produces a behavior embedding, which is compared to archetype centroids to compute distances. The minimum distance is converted to a confidence score via percentile rank. If confidence exceeds the threshold $\theta$, the binary gate enables hybrid scoring (combining LSTM and archetype scores). Otherwise, the gate disables archetype scoring, falling back to LSTM-only predictions.