Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Latent Concept-based Explanation of NLP Models

Xuemin Yu, Fahim Dalvi, Nadir Durrani, Marzia Nouri, Hassan Sajjad

TL;DR

This paper introduces LACOAT, a latent concept-based explanation method for NLP models that moves beyond word-level attributions by grounding predictions in latent concepts learned from training data. It consists of four modules—ConceptDiscoverer, PredictionAttributor, ConceptMapper, and PlausiFyer—that together discover latent concepts, identify salient representations, map them to latent concepts, and generate human-friendly explanations. Across POS, toxicity, sentiment, and MNLI tasks, LACOAT demonstrates strong faithfulness and utility, with robust last-layer explanations and meaningful human evaluations, and it enables analysis of how explanations evolve through model layers. The approach highlights a path toward faithful, interpretable AI by leveraging the training data's latent space to reveal how knowledge is structured inside DNNs, while acknowledging computational and data-size limitations and proposing future scalability improvements.

Abstract

Interpreting and understanding the predictions made by deep learning models poses a formidable challenge due to their inherently opaque nature. Many previous efforts aimed at explaining these predictions rely on input features, specifically, the words within NLP models. However, such explanations are often less informative due to the discrete nature of these words and their lack of contextual verbosity. To address this limitation, we introduce the Latent Concept Attribution method (LACOAT), which generates explanations for predictions based on latent concepts. Our foundational intuition is that a word can exhibit multiple facets, contingent upon the context in which it is used. Therefore, given a word in context, the latent space derived from our training process reflects a specific facet of that word. LACOAT functions by mapping the representations of salient input words into the training latent space, allowing it to provide latent context-based explanations of the prediction.

Latent Concept-based Explanation of NLP Models

TL;DR

This paper introduces LACOAT, a latent concept-based explanation method for NLP models that moves beyond word-level attributions by grounding predictions in latent concepts learned from training data. It consists of four modules—ConceptDiscoverer, PredictionAttributor, ConceptMapper, and PlausiFyer—that together discover latent concepts, identify salient representations, map them to latent concepts, and generate human-friendly explanations. Across POS, toxicity, sentiment, and MNLI tasks, LACOAT demonstrates strong faithfulness and utility, with robust last-layer explanations and meaningful human evaluations, and it enables analysis of how explanations evolve through model layers. The approach highlights a path toward faithful, interpretable AI by leveraging the training data's latent space to reveal how knowledge is structured inside DNNs, while acknowledging computational and data-size limitations and proposing future scalability improvements.

Abstract

Interpreting and understanding the predictions made by deep learning models poses a formidable challenge due to their inherently opaque nature. Many previous efforts aimed at explaining these predictions rely on input features, specifically, the words within NLP models. However, such explanations are often less informative due to the discrete nature of these words and their lack of contextual verbosity. To address this limitation, we introduce the Latent Concept Attribution method (LACOAT), which generates explanations for predictions based on latent concepts. Our foundational intuition is that a word can exhibit multiple facets, contingent upon the context in which it is used. Therefore, given a word in context, the latent space derived from our training process reflects a specific facet of that word. LACOAT functions by mapping the representations of salient input words into the training latent space, allowing it to provide latent context-based explanations of the prediction.
Paper Structure (68 sections, 13 figures, 26 tables)

This paper contains 68 sections, 13 figures, 26 tables.

Table of Contents

  1. Introduction
  2. Methodology
  3. ConceptDiscoverer
  4. PredictionAttributor
  5. ConceptMapper
  6. PlausiFyer
  7. Experimental Setup
  8. Data
  9. Models
  10. Module-specific hyperparameters
  11. Evaluation
  12. Qualitative Evaluation
  13. Evolution of Concepts
  14. Analyzing Last Layer Explanations
  15. Correct prediction with correct gold label
  16. ...and 53 more sections

Figures (13)

  • Figure 1: An example of various facets of word "trump"
  • Figure 2: Sentiment task: Latent concepts of the most attributed words in Layers 0, 6 and 12
  • Figure 3: A few examples of LACOAT explanations for BERT using POS and Sentiment tasks
  • Figure 4: Comparing explanation of XLMR (top) and RoBERTa (bottom)
  • Figure 5: Sentiment task: Examples of the latent concepts of the most attributed words in layers 0, 6 and 12
  • ...and 8 more figures