In-Context Learning Enhanced Credibility Transformer

TL;DR

The paper tackles the challenge of balancing predictive accuracy, explainability, and reliability in actuarial settings with limited history by marrying classical Bühlmann credibility with in-context learning. It introduces the In-context Learning enhanced Credibility Transformer (ICL-Credibility Transformer), incorporating an outcome token decorator, cross-batch attention, and a frozen decoder to enable context-driven adjustments within the CLS-token space. The authors establish theoretical links between ICL and credibility, demonstrate improved predictive performance on the French MTPL dataset, and show zero-shot generalization to unseen covariate levels. The approach yields data-driven credibility weighting via attention, enabling robust, context-aware predictions without full retraining, with implications for both pricing and reserving while calling for careful consideration of regulatory fairness and context selection. Overall, the work provides a principled, scalable framework for incorporating similar-risk information on-the-fly to improve actuarial predictions in dynamic portfolios.

Abstract

The starting point of our network architecture is the Credibility Transformer which extends the classical Transformer architecture by a credibility mechanism to improve model learning and predictive performance. This Credibility Transformer learns credibilitized CLS tokens that serve as learned representations of the original input features. In this paper we present a new paradigm that augments this architecture by an in-context learning mechanism, i.e., we increase the information set by a context batch consisting of similar instances. This allows the model to enhance the CLS token representations of the instances by additional in-context information and fine-tuning. We empirically verify that this in-context learning enhances predictive accuracy by adapting to similar risk patterns. Moreover, this in-context learning also allows the model to generalize to new instances which, e.g., have feature levels in the categorical covariates that have not been present when the model was trained -- for a relevant example, think of a new vehicle model which has just been developed by a car manufacturer.

Figures (6)

• Figure 1: ICL-Credibility Transformer architecture processing the target batch ${\cal B}_{\rm target}$ (yellow) and the context batch ${\cal B}_{\rm context}$ (blue) to form the prediction on the target set (orange).
• Figure 2: 2D PCA representation of the CLS tokens from the base Credibility Transformer (phase 1), before being used in the ICL model architecture. The background of the figure represents a color shade depending on the points in that region using a Voronoi-like tessellation.
• Figure 3: 2D PCA representation of the (2a) CLS tokens from the base Credibility Transformer after phase 2, (2b) decorated CLS tokens before the ICL mechanism and (2c) decorated CLS tokens after performing ICL augmentation.
• Figure 4: 2D PCA representation of the (3a) CLS tokens from the base Credibility Transformer after phase 3, (3b) decorated CLS tokens before the ICL mechanism and (3c) decorated CLS tokens after performing ICL augmentation.
• Figure 5: 2D PCA representation of the (4a) CLS tokens from the base Credibility Transformer after phase 2, (4b) decorated CLS tokens before the ICL mechanism and (4c) decorated CLS tokens after performing ICL augmentation, where the colors of the points represent the adjustment made on the log-scale to the base Credibility Transformer rate predictions by the ICL process.

Theorems & Definitions (2)

• Remark 2.1
• Proposition 3.1