ExplainableDetector: Exploring Transformer-based Language Modeling Approach for SMS Spam Detection with Explainability Analysis

TL;DR

This work addresses SMS spam detection by employing fine-tuned transformer models (DistilBERT and RoBERTa) augmented with explainability analyses. It tackles class imbalance through back-translation augmentation and benchmarks both ML baselines and transformer models on the UCI SMS Spam dataset, reporting RoBERTa achieving up to $99.84\%$ test accuracy on balanced data. Explainability is provided via LIME and Transformers Interpret to extract word-level attributions, improving transparency of predictions. The results demonstrate robust performance of transformer-based detectors alongside actionable interpretability, underscoring practical benefits for cybersecurity applications.

Abstract

SMS, or short messaging service, is a widely used and cost-effective communication medium that has sadly turned into a haven for unwanted messages, commonly known as SMS spam. With the rapid adoption of smartphones and Internet connectivity, SMS spam has emerged as a prevalent threat. Spammers have taken notice of the significance of SMS for mobile phone users. Consequently, with the emergence of new cybersecurity threats, the number of SMS spam has expanded significantly in recent years. The unstructured format of SMS data creates significant challenges for SMS spam detection, making it more difficult to successfully fight spam attacks in the cybersecurity domain. In this work, we employ optimized and fine-tuned transformer-based Large Language Models (LLMs) to solve the problem of spam message detection. We use a benchmark SMS spam dataset for this spam detection and utilize several preprocessing techniques to get clean and noise-free data and solve the class imbalance problem using the text augmentation technique. The overall experiment showed that our optimized fine-tuned BERT (Bidirectional Encoder Representations from Transformers) variant model RoBERTa obtained high accuracy with 99.84\%. We also work with Explainable Artificial Intelligence (XAI) techniques to calculate the positive and negative coefficient scores which explore and explain the fine-tuned model transparency in this text-based spam SMS detection task. In addition, traditional Machine Learning (ML) models were also examined to compare their performance with the transformer-based models. This analysis describes how LLMs can make a good impact on complex textual-based spam data in the cybersecurity field.

Figures (13)

• Figure 1: The methodology of SMS Spam Detection.
• Figure 2: The sample data overview
• Figure 3: Visualizing Imbalanced vs. Balanced Data Distribution.
• Figure 4: The Complete Fine-Tuning Process Flow.
• Figure 5: The complete approach to understanding model explainability