A Unified Framework with Novel Metrics for Evaluating the Effectiveness of XAI Techniques in LLMs

TL;DR

The paper presents elsarticle.cls, a LaTeX document class optimized for Elsevier journal submissions. It is built on top of article.cls to minimize package conflicts and to integrate smoothly with widely used tools like natbib and hyperref, while offering robust frontmatter support and flexible formatting options. The authors detail the key differences from the previous elsart.cls release and provide practical installation instructions via CTAN, including building the class from source and placing it in the TEXMF hierarchy. This framework simplifies compliant, conflict-free manuscript preparation and enhances compatibility across LaTeX configurations common in scientific publishing.

Abstract

The increasing complexity of LLMs presents significant challenges to their transparency and interpretability, necessitating the use of eXplainable AI (XAI) techniques to enhance trustworthiness and usability. This study introduces a comprehensive evaluation framework with four novel metrics for assessing the effectiveness of five XAI techniques across five LLMs and two downstream tasks. We apply this framework to evaluate several XAI techniques LIME, SHAP, Integrated Gradients, Layer-wise Relevance Propagation (LRP), and Attention Mechanism Visualization (AMV) using the IMDB Movie Reviews and Tweet Sentiment Extraction datasets. The evaluation focuses on four key metrics: Human-reasoning Agreement (HA), Robustness, Consistency, and Contrastivity. Our results show that LIME consistently achieves high scores across multiple LLMs and evaluation metrics, while AMV demonstrates superior Robustness and near-perfect Consistency. LRP excels in Contrastivity, particularly with more complex models. Our findings provide valuable insights into the strengths and limitations of different XAI methods, offering guidance for developing and selecting appropriate XAI techniques for LLMs.