Mechanistic interpretability of large language models with applications to the financial services industry

TL;DR

This paper investigates GPT-2 Small’s attention pattern when prompted to identify potential violation of Fair Lending laws, and pioneering the use of mechanistic interpretability to shed some light on the inner workings of large language models for use in financial services applications.

Abstract

Large Language Models such as GPTs (Generative Pre-trained Transformers) exhibit remarkable capabilities across a broad spectrum of applications. Nevertheless, due to their intrinsic complexity, these models present substantial challenges in interpreting their internal decision-making processes. This lack of transparency poses critical challenges when it comes to their adaptation by financial institutions, where concerns and accountability regarding bias, fairness, and reliability are of paramount importance. Mechanistic interpretability aims at reverse engineering complex AI models such as transformers. In this paper, we are pioneering the use of mechanistic interpretability to shed some light on the inner workings of large language models for use in financial services applications. We offer several examples of how algorithmic tasks can be designed for compliance monitoring purposes. In particular, we investigate GPT-2 Small's attention pattern when prompted to identify potential violation of Fair Lending laws. Using direct logit attribution, we study the contributions of each layer and its corresponding attention heads to the logit difference in the residual stream. Finally, we design clean and corrupted prompts and use activation patching as a causal intervention method to localize our task completion components further. We observe that the (positive) heads $10.2$ (head $2$, layer $10$), $10.7$, and $11.3$, as well as the (negative) heads $9.6$ and $10.6$ play a significant role in the task completion.

Figures (21)

• Figure 1: The transformer architecture. Picture adapted from vaswani2017attention.
• Figure 2: Curve detector and line detector neurons in early layers form "full curve detector circuits" that can be used to create complex shapes and geometry. The picture is based on experiments with the https://ieeexplore.ieee.org/document/7298594 convolutional architecture. It is from olah2020zoom (taken from https://github.com/distillpub/post--circuits-zoom-in under the CC-BY 4.0 license).
• Figure 3: An illustration of polysemantic neurons in the context of vision models adapted from olah2020zoom (taken from https://github.com/distillpub/post--circuits-zoom-in under the CC-BY 4.0 license).
• Figure 4: A depiction of the residual stream in a transformer (courtesy of elhage2021mathematical).
• Figure 5: Top, an illustration of QK (query-key) and OV (output-value) circuits (courtesy of elhage2021mathematical). Bottom, an illustration of induction heads' behavior (courtesy of olsson2022context).