Weapons of Online Harassment: Menacing and Profiling Users via Social Apps
Sanjana Cheerla, Vaibhav Garg, Saikath Bhattacharya, Munindar P. Singh
TL;DR
This work treats mobile social apps as sociotechnical systems and investigates technology-facilitated online harassment as reported in user reviews. It constructs a large-scale dataset with over $3.0$ million reviews across about $1{,}800$ apps and trains a high-recall $XLNet$-based classifier to detect two harassment forms, Menacing and Profiling, achieving $90\%$ and $85\%$ recall respectively. Key findings show Profiling is more prevalent than Menacing, with abusers more often female in Profiling and distinct victim emotions (anger, disgust, fear) linked to harassment; the patterns vary by app category, especially dating/chat apps. The study identifies $1{,}395$ harassment-bearing apps and notifies the top $48$ developers, offering concrete guidance for developers and platform stores to mitigate abuse while acknowledging limitations and proposing avenues for multilingual and broader-app analyses.
Abstract
Viewing social apps as sociotechnical systems makes clear that they are not mere pieces of technology but mediate human interaction and may unintentionally enable harmful behaviors like online harassment. As more users interact through social apps, instances of harassment increase. We observed that app reviews often describe harassment. Accordingly, we built a dataset of over 3 million reviews and 1,800 apps. We discovered that two forms of harassment are prevalent, Menacing and Profiling. We built a computational model for identifying reviews indicating harassment, achieving high recalls of 90% for Menacing and 85% for Profiling. We analyzed the data further to better understand the terrain of harassment. Surprisingly, abusers most often have female identities. Also, what distinguishes negative from neutral reviews is the greater prevalence of anger, disgust, and fear. Applying our model, we identified 1,395 apps enabling harassment and notified developers of the top 48 with the highest user-reported harassment.