The Credibility Transformer

TL;DR

This work introduces a novel credibility mechanism based on a special token that should be seen as an encoder that consists of a credibility weighted average of prior information and observation based information that leads to predictive models that are superior to state-of-the-art deep learning models.

Abstract

Inspired by the large success of Transformers in Large Language Models, these architectures are increasingly applied to tabular data. This is achieved by embedding tabular data into low-dimensional Euclidean spaces resulting in similar structures as time-series data. We introduce a novel credibility mechanism to this Transformer architecture. This credibility mechanism is based on a special token that should be seen as an encoder that consists of a credibility weighted average of prior information and observation based information. We demonstrate that this novel credibility mechanism is very beneficial to stabilize training, and our Credibility Transformer leads to predictive models that are superior to state-of-the-art deep learning models.

Figures (13)

• Figure 1: A diagram showing the augmented input tensor $\boldsymbol{x}^+_{1:T+1}$ consisting of input and positional encoding and the CLS token, moreover, it shows how the scalar components comprising the CLS token encode the columns of the input tensor $\boldsymbol{x}_{1:T}$.
• Figure 2: Diagram of how the attention mechanism can be viewed as a linear credibility formula.
• Figure 3: (lhs) Scatterplot of predictions from one run of the Credibility Transformer (CT) vs. GLM predictions; (rhs) density of log-differences of the two predictors, the vertical dotted lines correspond to 2 empirical standard deviations.
• Figure 4: Scatterplot of ensemble Credibility Transformer predictions: NormFormer SGD fitting against nadam fitting.
• Figure 5: Visualization of differentiable Piecewise Linear Encoding (PLE) for a numerical feature.