Homograph Attacks on Maghreb Sentiment Analyzers

TL;DR

The paper addresses the vulnerability of sentiment analysis for Maghrebi dialects to homograph-based Unicode perturbations, particularly in Arabizi. It applies a perturbation where up to 90% of Latin characters are replaced with Unicode homographs on test data, while training data remains unchanged, and fine-tunes several dialect-focused SA models with a learning rate of $2e^{-5}$ for 3 epochs. Results reveal dramatic F1 declines, most pronounced on Arabizi-only datasets (e.g., up to about 65%), with TunBERT on Tunizi2020 dropping from 0.95 to 0.33, and mixed-script datasets showing somewhat smaller decreases around 24.7%. The work highlights ethical considerations and the need for defense mechanisms, proposing an initial protective input layer to detect and correct perturbations as a foundation for more robust, responsible NLP in low-resource Maghrebi contexts.

Abstract

We examine the impact of homograph attacks on the Sentiment Analysis (SA) task of different Arabic dialects from the Maghreb North-African countries. Homograph attacks result in a 65.3% decrease in transformer classification from an F1-score of 0.95 to 0.33 when data is written in "Arabizi". The goal of this study is to highlight LLMs weaknesses' and to prioritize ethical and responsible Machine Learning.