Discretized Integrated Gradients for Explaining Language Models

TL;DR

This paper addresses the interpretability challenge of language models by improving gradient-based explanations for discrete text data. It introduces Discretized Integrated Gradients (DIG), which replaces IG's straight-line interpolation with monotone, non-linear interpolation in the word embedding space, guided by anchors from the vocabulary. Two interpolation strategies, DIG-Greedy and DIG-MaxCount, generate interpolation points that remain close to real word embeddings, reducing word-approximation error and yielding more faithful attributions. Across three datasets and three language models, DIG demonstrates superior automated metrics and gains in human trust relative to IG and other baselines. The work includes ablations, analysis of path density, and release of the DIG code to support reproducibility.

Abstract

As a prominent attribution-based explanation algorithm, Integrated Gradients (IG) is widely adopted due to its desirable explanation axioms and the ease of gradient computation. It measures feature importance by averaging the model's output gradient interpolated along a straight-line path in the input data space. However, such straight-line interpolated points are not representative of text data due to the inherent discreteness of the word embedding space. This questions the faithfulness of the gradients computed at the interpolated points and consequently, the quality of the generated explanations. Here we propose Discretized Integrated Gradients (DIG), which allows effective attribution along non-linear interpolation paths. We develop two interpolation strategies for the discrete word embedding space that generates interpolation points that lie close to actual words in the embedding space, yielding more faithful gradient computation. We demonstrate the effectiveness of DIG over IG through experimental and human evaluations on multiple sentiment classification datasets. We provide the source code of DIG to encourage reproducible research.

Figures (6)

• Figure 1: An illustration of paths used in IG and DIG. IG uses a straight line interpolation with points as depicted by green squares. In contrast, DIG uses a non-linear path (shown in blue) with interpolation points (red stars) lying close to words in the embedding space.
• Figure 2: Overview of paths used in DIG and IG. The gray region is the neighborhood of $w$. Green line depicts the straight-line path used by IG. Left: In DIG-Greedy, we first monotonize each word in the neighborhood (red arrow) and the word closest to its corresponding monotonic point is selected as the anchor ($w_5$ since the red arrow of $w_5$ has the smallest magnitude). Right: In DIG-MaxCount we select the word with the highest number of monotonic dimensions (count shown in $[.]$) as the anchor word ($w_4$), followed by changing the non-monotonic dimensions of $w_4$ (red arrow to $c$). Repeating this iteratively gives the non-linear blue path for DIG with the red stars as interpolation points. Please refer to Section \ref{['sec:dig']} for more details. Figure best viewed in color.
• Figure 3: Result of human evaluation on DistilBERT model fine-tuned on SST2 dataset and BERT model fine-tuned on Rotten Tomatoes dataset. A lower mean rank means higher trustworthy explanation algorithm. For more details, refer to Section \ref{['sec:human']}
• Figure 4: Effect of increasing number of interpolation points $m$ on IG and DIG.
• Figure 5: Effect of changing top-k% in log-odds, comprehensiveness, and sufficiency metric for the DistilBERT model fine-tuned on SST2 dataset.