Dark patterns in e-commerce: a dataset and its baseline evaluations
Yuki Yada, Jiaying Feng, Tsuneo Matsumoto, Nao Fukushima, Fuyuko Kido, Hayato Yamana
TL;DR
This work tackles the lack of labeled data for detecting dark patterns in e-commerce and their privacy implications by constructing a balanced dataset and establishing strong baselines. Starting from Mathur et al.'s 11kScale dark-pattern texts, the authors derive 1,178 positive samples and generate 1,178 negative samples from the same sites, totaling 2,356 texts, and they validate a text-segmentation pipeline to extract UI-relevant content. They compare classical NLP baselines and transformer-based models, with RoBERTa_large achieving the peak accuracy of 0.975 under 5-fold cross-validation, establishing a solid benchmark for future work. The dataset and code are publicly available, enabling broader research into automatic dark-pattern detection and paving the way for incorporating non-text UX cues in future analyses.
Abstract
Dark patterns, which are user interface designs in online services, induce users to take unintended actions. Recently, dark patterns have been raised as an issue of privacy and fairness. Thus, a wide range of research on detecting dark patterns is eagerly awaited. In this work, we constructed a dataset for dark pattern detection and prepared its baseline detection performance with state-of-the-art machine learning methods. The original dataset was obtained from Mathur et al.'s study in 2019, which consists of 1,818 dark pattern texts from shopping sites. Then, we added negative samples, i.e., non-dark pattern texts, by retrieving texts from the same websites as Mathur et al.'s dataset. We also applied state-of-the-art machine learning methods to show the automatic detection accuracy as baselines, including BERT, RoBERTa, ALBERT, and XLNet. As a result of 5-fold cross-validation, we achieved the highest accuracy of 0.975 with RoBERTa. The dataset and baseline source codes are available at https://github.com/yamanalab/ec-darkpattern.