PANTHER: Generative Pretraining Beyond Language for Sequential User Behavior Modeling

TL;DR

PANTHER introduces a generative pretraining paradigm for sequential user behaviors, addressing the unique challenges of high-cardinality transactional data and real-time production latency. The framework combines structured tokenization, a Sequence Pattern Recognition Module that captures local periodicities with prototype-driven alignment, and a user-profile embedding learned via contrastive objectives to produce transferable representations. It deploys offline pretrained embeddings in a real-time hybrid inference pipeline, enabling millisecond-scale scoring for fraud detection and personalized recommendations. Across WeChat Pay and public benchmarks, PANTHER achieves substantial gains in next-transaction prediction, fraud recall, and cross-domain transfer, demonstrating strong generalization and practical impact in industrial sequential decision-making.

Abstract

Large language models (LLMs) have shown that generative pretraining can distill vast world knowledge into compact token representations. While LLMs encapsulate extensive world knowledge, they remain limited in modeling the behavioral knowledge contained within user interaction histories. User behavior forms a distinct modality, where each action, defined by multi-dimensional attributes such as time, context, and transaction type, constitutes a behavioral token. Modeling these high-cardinality sequences is challenging, and discriminative models often falter under limited supervision. To bridge this gap, we extend generative pretraining to user behavior, learning transferable representations from unlabeled behavioral data analogous to how LLMs learn from text. We present PANTHER, a hybrid generative-discriminative framework that unifies user behavior pretraining and downstream adaptation, enabling large-scale sequential user representation learning and real-time inference. PANTHER introduces: (1) Structured Tokenization to compress multi-dimensional transaction attributes into an interpretable vocabulary; (2) Sequence Pattern Recognition Module (SPRM) for modeling periodic transaction motifs; (3) a Unified User-Profile Embedding that fuses static demographics with dynamic transaction histories; and (4) Real-time scalability enabled by offline caching of pretrained embeddings for millisecond-level inference. Fully deployed and operational online at WeChat Pay, PANTHER delivers a 25.6 percent boost in next-transaction prediction HitRate@1 and a 38.6 percent relative improvement in fraud detection recall over baselines. Cross-domain evaluations on public benchmarks show strong generalization, achieving up to 21 percent HitRate@1 gains over transformer baselines, establishing PANTHER as a scalable, high-performance framework for industrial sequential user behavior modeling.

Figures (4)

• Figure 1: Illustration of the periodic pattern of user behaviors
• Figure 2: Key components of PANTHER: Structured Tokenization, SPRM and user profile embedding
• Figure 3: Demonstration of the cross attention score between recurring three-gram user payment behaviors and the learnable pattern prototypes.
• Figure 4: Fraud recall improvement at operational thresholds (Recall@Top-K).