Adversarial Attacks in NLP for Abuse Detection Systems

This paper examines the weaknesses of machine learning models to adversarial attacks in online abuse detection. With the growth of user- generated content online, platforms rely on automated systems to detect and filter harmful content at scale. However, these systems remain vulnerable to manipulations by bad actors designed to circumvent detection. We investigate two prominent attack strategies TextFooler and HotFlip against transformer-based models trained on the Jigsaw Toxic Comment Classification dataset. Our experiments reveal considerable degradation in model performance under attack conditions, with accuracy drops of approximately 20%. This paper provides a detailed analysis of these attack strategies, implementation methods, and their impact on model reliability. The findings highlight critical vulnerabilities in current abuse detection systems and demonstrate the need for more robust approaches to maintain platform safety and integrity.