o-MEGA: Optimized Methods for Explanation Generation and Analysis

TL;DR

The paper tackles the challenge of selecting and configuring explainability methods for transformer-based semantic matching, particularly in claim matching for fact-checking. It introduces o-mega, a hyperparameter-optimization framework built on Optuna and leveraging Captum-based XAI techniques to automatically identify effective explanations. Through a case study on post-claim matching with the MultiClaim dataset, it demonstrates that Occlusion offers the best balance of fidelity and plausibility, while TPESampler provides efficient optimization, enabling practical deployment of interpretable semantic matching. Overall, o-mega advances trustworthy AI in misinformation detection by automating the discovery of domain-appropriate explanations and their configurations, reducing manual experimentation and enhancing user trust.

Abstract

The proliferation of transformer-based language models has revolutionized NLP domain while simultaneously introduced significant challenges regarding model transparency and trustworthiness. The complexity of achieving explainable systems in this domain is evidenced by the extensive array of explanation methods and evaluation metrics developed by researchers. To address the challenge of selecting optimal explainability approaches, we present \textbf{\texttt{o-mega}}, a hyperparameter optimization tool designed to automatically identify the most effective explainable AI methods and their configurations within the semantic matching domain. We evaluate o-mega on a post-claim matching pipeline using a curated dataset of social media posts paired with refuting claims. Our tool systematically explores different explainable methods and their hyperparameters, demonstrating improved transparency in automated fact-checking systems. As a result, such automated optimization of explanation methods can significantly enhance the interpretability of claim-matching models in critical applications such as misinformation detection, contributing to more trustworthy and transparent AI systems.

Figures (7)

• Figure 1: The architecture of the o-mega tool.
• Figure 2: Configuration block specifying the base parameters for hyperoptimization and evaluation of explanations
• Figure 3: Configuration block for Optuna Sampler and its parameters.
• Figure 4: Configuration block specifying hyperparameters for explainable method.
• Figure 5: One example of the post-claim pair with annotated explanations.