Challenges and Opportunities in Text Generation Explainability

TL;DR

This paper addresses the explainability of text generation by focusing on attribution-based explanations for autoregressive language models. It identifies 17 challenges across dataset creation, explanation design, and evaluation, and argues for a holistic, human-in-the-loop framework that involves stakeholders from the start. The authors propose well-designed perturbed datasets and probabilistic word-level explanations as pathways to robust xAI benchmarking in NLP. Overall, the work provides a roadmap for developing, evaluating, and comparing explainability methods for text generation and highlights opportunities to advance practical interpretability in real-world systems.

Abstract

The necessity for interpretability in natural language processing (NLP) has risen alongside the growing prominence of large language models. Among the myriad tasks within NLP, text generation stands out as a primary objective of autoregressive models. The NLP community has begun to take a keen interest in gaining a deeper understanding of text generation, leading to the development of model-agnostic explainable artificial intelligence (xAI) methods tailored to this task. The design and evaluation of explainability methods are non-trivial since they depend on many factors involved in the text generation process, e.g., the autoregressive model and its stochastic nature. This paper outlines 17 challenges categorized into three groups that arise during the development and assessment of attribution-based explainability methods. These challenges encompass issues concerning tokenization, defining explanation similarity, determining token importance and prediction change metrics, the level of human intervention required, and the creation of suitable test datasets. The paper illustrates how these challenges can be intertwined, showcasing new opportunities for the community. These include developing probabilistic word-level explainability methods and engaging humans in the explainability pipeline, from the data design to the final evaluation, to draw robust conclusions on xAI methods.

Figures (5)

• Figure 1: Challenges at various stages of the text generation explainability process, including the dataset creation, the explanation design, and the explanation evaluation. Different stakeholders address these challenges. Linguists are responsible for designing semantic, syntactic, and grammatical perturbations. End-users contribute to defining the overall format of expected explanations, e.g., size of explanations. Machine learning (ML) practitioners tackle the remaining challenges related to the language model, the data, and the evaluation metrics.
• Figure 2: Example of an explanation pipeline for text generation. Challenges related to the dataset creation are referred to by icon D, and the ones related to the explanation design with icon X. The autoregressive language model that includes the tokenizer, the decoder, and the sampling method, takes as input an unfinished sentence $X$ and generates the next token $Y$. The explainability method then produces an explanation $E$ that is transformed into a textual explanation $E^{\%}$ after a step of thresholding and word exclusion.
• Figure 3: Evaluation procedure with well-designed perturbations. Different types of perturbations are used to create tailored datasets. The selected perturbation $U$ affects the input sentence $X$ and the explanation $E$ generated by the xAI method. By assigning linguistic properties to the different types of perturbation, it is possible to use the results of the evaluation phase to characterize xAI methods. Icons Ddescribe challenges related to the dataset creation and icons Edescribe challenges in the evaluation phase of explanations.
• Figure 4: Quantitative evaluation of explainability employs three key quality metrics. While accuracy and faithfulness assess explanations at the instance level, coherency adopts a contrastive approach, evaluating pairs of sentences, including an input sentence and its perturbed counterpart. Challenges are inherent to each metric and are represented by the icon E.
• Figure 5: Summary of the challenges in explainability for text generation. Challenges arise in the dataset creation, the explanation design and the evaluation to rigorously characterize explainability methods.