Designing an attack-defense game: how to increase robustness of financial transaction models via a competition

TL;DR

A novel type of competition that allows a realistic and detailed investigation of problems in financial transaction data, with participants directly oppose each other, proposing attacks and defenses — so they are examined in close-to-real-life conditions.

Abstract

Banks routinely use neural networks to make decisions. While these models offer higher accuracy, they are susceptible to adversarial attacks, a risk often overlooked in the context of event sequences, particularly sequences of financial transactions, as most works consider computer vision and NLP modalities. We propose a thorough approach to studying these risks: a novel type of competition that allows a realistic and detailed investigation of problems in financial transaction data. The participants directly oppose each other, proposing attacks and defenses -- so they are examined in close-to-real-life conditions. The paper outlines our unique competition structure with direct opposition of participants, presents results for several different top submissions, and analyzes the competition results. We also introduce a new open dataset featuring financial transactions with credit default labels, enhancing the scope for practical research and development.

Figures (8)

• Figure 1: Competition scheme with attack and defense tracks and pre-tournament and tournament phases. Better to view in zoom
• Figure 2: On the left pictures \ref{['fig:attack_scores']} and \ref{['fig:defense_scores']} solid lines are empirical cumulative density functions for different stages of the attack \ref{['fig:attack_scores']} and defense \ref{['fig:defense_scores']} tracks of the competition. Dashed lines are the top score for each phase (1 or 2) and baseline scores. Higher ROC AUC differences are better for an attack. Bigger Harmonic means of ROC AUCs are better for defence. On the right pictures \ref{['fig:attack_dynamic']} and \ref{['fig:defense_dynamic']} dynamic of the top private score for attack \ref{['fig:attack_dynamic']} and defense \ref{['fig:defense_dynamic']} competition is presented. These scores were hidden from participants until the end of the competition. With a blue curve, we highlight the top achieved score at a given moment in time, and each point corresponds to a score for a single submission. Better to view in zoom.
• Figure 3: ROC AUC decreases for pairs of attack and defense from the competition stage. We removed unfair scores for pairs when the attack and defense model authors coincided and put instead white squares.
• Figure 4: Data split structure
• Figure 5: Scheme of the attacked GRU model