Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Uniting contrastive and generative learning for event sequences models

Aleksandr Yugay, Alexey Zaytsev

TL;DR

The paper addresses the need for universal representations of transactional sequences that capture both local (current-state) and global (behavioral-pattern) information in banking data. It introduces two self-supervised learning approaches—Contrastive Masked Language Modeling (CMLM) and CoLES—and proposes a simple hybrid objective that combines them to produce embeddings with balanced local and global properties. Through experiments on four public financial datasets for sequence classification and next-event prediction, the authors show that CMLM improves local pattern modeling, CoLES strengthens global representations, and the combined CMLM+CoLES approach achieves gains across both tasks, highlighting the synergy of the two paradigms. The proposed framework offers a robust, scalable method for event sequence representation learning in finance, with practical implications for risk management, churn prediction, and personalized offers.

Abstract

High-quality representation of transactional sequences is vital for modern banking applications, including risk management, churn prediction, and personalized customer offers. Different tasks require distinct representation properties: local tasks benefit from capturing the client's current state, while global tasks rely on general behavioral patterns. Previous research has demonstrated that various self-supervised approaches yield representations that better capture either global or local qualities. This study investigates the integration of two self-supervised learning techniques - instance-wise contrastive learning and a generative approach based on restoring masked events in latent space. The combined approach creates representations that balance local and global transactional data characteristics. Experiments conducted on several public datasets, focusing on sequence classification and next-event type prediction, show that the integrated method achieves superior performance compared to individual approaches and demonstrates synergistic effects. These findings suggest that the proposed approach offers a robust framework for advancing event sequences representation learning in the financial sector.

Uniting contrastive and generative learning for event sequences models

TL;DR

The paper addresses the need for universal representations of transactional sequences that capture both local (current-state) and global (behavioral-pattern) information in banking data. It introduces two self-supervised learning approaches—Contrastive Masked Language Modeling (CMLM) and CoLES—and proposes a simple hybrid objective that combines them to produce embeddings with balanced local and global properties. Through experiments on four public financial datasets for sequence classification and next-event prediction, the authors show that CMLM improves local pattern modeling, CoLES strengthens global representations, and the combined CMLM+CoLES approach achieves gains across both tasks, highlighting the synergy of the two paradigms. The proposed framework offers a robust, scalable method for event sequence representation learning in finance, with practical implications for risk management, churn prediction, and personalized offers.

Abstract

High-quality representation of transactional sequences is vital for modern banking applications, including risk management, churn prediction, and personalized customer offers. Different tasks require distinct representation properties: local tasks benefit from capturing the client's current state, while global tasks rely on general behavioral patterns. Previous research has demonstrated that various self-supervised approaches yield representations that better capture either global or local qualities. This study investigates the integration of two self-supervised learning techniques - instance-wise contrastive learning and a generative approach based on restoring masked events in latent space. The combined approach creates representations that balance local and global transactional data characteristics. Experiments conducted on several public datasets, focusing on sequence classification and next-event type prediction, show that the integrated method achieves superior performance compared to individual approaches and demonstrates synergistic effects. These findings suggest that the proposed approach offers a robust framework for advancing event sequences representation learning in the financial sector.
Paper Structure (14 sections, 5 equations, 3 figures, 4 tables)

This paper contains 14 sections, 5 equations, 3 figures, 4 tables.

Table of Contents

  1. Introduction
  2. Related work
  3. Methods
  4. Problem statement
  5. Self-supervised Learning (SSL).
  6. Model architecture
  7. Contrastive learning with CoLES
  8. Our CMLM
  9. Hybrid approach: CMLM + CoLES
  10. Experiments
  11. Datasets
  12. Results
  13. Conclusions
  14. Acknowledgments

Figures (3)

  • Figure 1: Diagram of a neural network model showing the transformation from raw transactions to contextualized transaction embeddings.
  • Figure 2: Scheme of our CoLES+CMLM approach
  • Figure 3: Quality metrics for global and local embedding tasks, showing ROC-AUC scores across four datasets. Each point represents the mean performance of a model, with error bars indicating standard deviation.