More Questions than Answers? Lessons from Integrating Explainable AI into a Cyber-AI Tool
Ashley Suh, Harry Li, Caitlin Kenney, Kenneth Alperin, Steven R. Gomez
TL;DR
The paper investigates integrating Explainable AI into a cyber-operations workflow by applying SHAP and LIME to a source-code classifier and embedding the explanations in a decision-support tool. It finds that post-hoc, local explanations are often hard for non-experts to interpret, and disagreements between explanation methods can undermine trust. The authors argue for higher-level, domain-tailored visualizations and discuss potential remedies, including dialogue-assisted explanations and emerging LLMs, while warning about hallucinations. The work highlights practical gaps in XAI for cybersecurity and suggests directions to make explanations more usable in real-time operator workflows.
Abstract
We share observations and challenges from an ongoing effort to implement Explainable AI (XAI) in a domain-specific workflow for cybersecurity analysts. Specifically, we briefly describe a preliminary case study on the use of XAI for source code classification, where accurate assessment and timeliness are paramount. We find that the outputs of state-of-the-art saliency explanation techniques (e.g., SHAP or LIME) are lost in translation when interpreted by people with little AI expertise, despite these techniques being marketed for non-technical users. Moreover, we find that popular XAI techniques offer fewer insights for real-time human-AI workflows when they are post hoc and too localized in their explanations. Instead, we observe that cyber analysts need higher-level, easy-to-digest explanations that can offer as little disruption as possible to their workflows. We outline unaddressed gaps in practical and effective XAI, then touch on how emerging technologies like Large Language Models (LLMs) could mitigate these existing obstacles.