The value of manual annotation in assessing trends of hate speech on social media: was antisemitism on the rise during the tumultuous weeks of Elon Musk’s Twitter takeover?

In recent years, there has been a growing interest in research on hate speech on social media. However, researchers face many challenges in producing meaningful results and must develop innovative methods to keep pace with the rapidly evolving nature of this field. How can we effectively determine the prevalence of specific types of hate speech on a given platform? What approaches can we employ to assess whether there has been an increase or decrease in content that can be classified as hate speech? Using the context of Elon Musk's acquisition of Twitter, a period characterized by media reports suggesting a surge in antisemitism on the platform, we explore a range of qualitative and quantitative computational methods. Our analysis reveals that, starting from October 9, 2022, the usage of the term “Jews” on Twitter nearly doubled compared to the preceding period. Additionally, there was a sudden spike in the use of the term “K***s.” However, the question arises: how indicative are these trends of a rise in antisemitism on that platform? We demonstrate that relying solely on keyword-based timelines can be misleading. Nevertheless, when utilized alongside corroborating timelines incorporating additional keywords identified through word frequency analysis, they can serve as powerful tools for content estimation. Nonetheless, it is crucial to supplement these approaches with interpretative methods to validate assumptions based on timelines. By employing a triangulation of methods encompassing descriptive analysis, such as timelines, word and retweet frequency analysis, and manual interpretation and labeling of representative samples, we uncover that discussions about Jews on Twitter during a turbulent 5-week period were predominantly centered around antisemitism. However, these discussions took various forms, including expressing concerns about the increase in antisemitism, denouncing antisemitism, remembering the Holocaust, refuting accusations of antisemitism, and even promoting antisemitic ideologies. We observe a significant escalation in both the volume and the proportion of antisemitic tropes within these conversations, particularly evident in late October 2022. This increase can be attributed to three triggering events that might have been overlooked when drawing conclusions solely from a simplistic timeline analysis of the term “Jews” or targeted slurs. In conclusion, we advocate for a mixed-methods approach where quantitative computational tools are complemented by qualitative discourse analysis. This combination is essential for comprehensively examining trends in complex content such as hate speech. Additionally, the integration of “manual” observation and labeling of representative data samples proves particularly valuable for distinguishing between instances of hate speech and the act of calling out such speech.