Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Learning Intrinsic Dimension via Information Bottleneck for Explainable Aspect-based Sentiment Analysis

Zhenxiao Cheng, Jie Zhou, Wen Wu, Qin Chen, Liang He

TL;DR

This paper addresses the uneven importance of embedding dimensions in Aspect-based Sentiment Analysis by introducing IBG, a gradient-based explanation framework that learns a compact intrinsic dimension via an Information Bottleneck. It introduces the iBiL module to compress word embeddings to a low-dimensional intrinsic space while keeping the original embeddings fixed, and combines gradient signals from both spaces to produce sentiment-aware explanations. Extensive experiments on four ABSA datasets show that IBG improves both predictive performance and interpretability over strong baselines and standard explanation methods; ablation studies confirm the contributions of the iBiL and IB components, and analyses identify effective intrinsic sizes around 10–20 dimensions. The framework is model-agnostic and demonstrates potential for broader application to ABSA tasks and future work on large-scale language models, offering practical benefits for explainability and efficiency in NLP systems.

Abstract

Gradient-based explanation methods are increasingly used to interpret neural models in natural language processing (NLP) due to their high fidelity. Such methods determine word-level importance using dimension-level gradient values through a norm function, often presuming equal significance for all gradient dimensions. However, in the context of Aspect-based Sentiment Analysis (ABSA), our preliminary research suggests that only specific dimensions are pertinent. To address this, we propose the Information Bottleneck-based Gradient (\texttt{IBG}) explanation framework for ABSA. This framework leverages an information bottleneck to refine word embeddings into a concise intrinsic dimension, maintaining essential features and omitting unrelated information. Comprehensive tests show that our \texttt{IBG} approach considerably improves both the models' performance and interpretability by identifying sentiment-aware features.

Learning Intrinsic Dimension via Information Bottleneck for Explainable Aspect-based Sentiment Analysis

TL;DR

This paper addresses the uneven importance of embedding dimensions in Aspect-based Sentiment Analysis by introducing IBG, a gradient-based explanation framework that learns a compact intrinsic dimension via an Information Bottleneck. It introduces the iBiL module to compress word embeddings to a low-dimensional intrinsic space while keeping the original embeddings fixed, and combines gradient signals from both spaces to produce sentiment-aware explanations. Extensive experiments on four ABSA datasets show that IBG improves both predictive performance and interpretability over strong baselines and standard explanation methods; ablation studies confirm the contributions of the iBiL and IB components, and analyses identify effective intrinsic sizes around 10–20 dimensions. The framework is model-agnostic and demonstrates potential for broader application to ABSA tasks and future work on large-scale language models, offering practical benefits for explainability and efficiency in NLP systems.

Abstract

Gradient-based explanation methods are increasingly used to interpret neural models in natural language processing (NLP) due to their high fidelity. Such methods determine word-level importance using dimension-level gradient values through a norm function, often presuming equal significance for all gradient dimensions. However, in the context of Aspect-based Sentiment Analysis (ABSA), our preliminary research suggests that only specific dimensions are pertinent. To address this, we propose the Information Bottleneck-based Gradient (\texttt{IBG}) explanation framework for ABSA. This framework leverages an information bottleneck to refine word embeddings into a concise intrinsic dimension, maintaining essential features and omitting unrelated information. Comprehensive tests show that our \texttt{IBG} approach considerably improves both the models' performance and interpretability by identifying sentiment-aware features.
Paper Structure (23 sections, 11 equations, 7 figures, 2 tables)

This paper contains 23 sections, 11 equations, 7 figures, 2 tables.

Table of Contents

  1. Introduction
  2. Preliminary Analysis
  3. Our Approach
  4. Overview of IBG
  5. Information Bottleneck-based Intrinsic Learning
  6. Experiment Setups
  7. Datasets, Metrics and Settings
  8. Datasets.
  9. Metrics.
  10. Settings.
  11. Baselines
  12. Results and Analyses
  13. Main Results
  14. Ablation Studies
  15. Further Analysis
  16. ...and 8 more sections

Figures (7)

  • Figure 1: IBG compresses the 100 noisy dimensions of token gradient into 8 intrinsic dimensions via information bottleneck.
  • Figure 2: Visualization of the gradients on hidden dimension over four classic ABSA datasets.
  • Figure 3: The influence of top k hidden dimension
  • Figure 4: Percentage of Samples Where Each Dimension is in the Top 100 Importance
  • Figure 5: The framework of our IBG approach.
  • ...and 2 more figures