LLM4ES: Learning User Embeddings from Event Sequences via Large Language Models
Aleksei Shestov, Omar Zoloev, Maksim Makarenko, Mikhail Orlov, Egor Fadeev, Ivan Kireev, Andrey Savchenko
TL;DR
LLM4ES introduces a text-enrichment pipeline that converts event sequences into descriptive text, uses a pre-trained LLM to enrich this representation, and fine-tunes the model to extract rich user embeddings via next-token prediction. The approach achieves state-of-the-art ROC-AUC on financial transaction datasets and generalizes to non-financial domains like MovieLens, with notable gains when embedded alongside existing methods in ensembles. Ablation studies substantiate the contributions of text enrichment, full fine-tuning, and cross-dataset training, while the framework demonstrates robustness across data regimes. Overall, LLM4ES offers a versatile, domain-agnostic method for learning user embeddings from sequential event data, with broad applicability from finance to healthcare and beyond.
Abstract
This paper presents LLM4ES, a novel framework that exploits large pre-trained language models (LLMs) to derive user embeddings from event sequences. Event sequences are transformed into a textual representation, which is subsequently used to fine-tune an LLM through next-token prediction to generate high-quality embeddings. We introduce a text enrichment technique that enhances LLM adaptation to event sequence data, improving representation quality for low-variability domains. Experimental results demonstrate that LLM4ES achieves state-of-the-art performance in user classification tasks in financial and other domains, outperforming existing embedding methods. The resulting user embeddings can be incorporated into a wide range of applications, from user segmentation in finance to patient outcome prediction in healthcare.