Predicting zip code-level vaccine hesitancy in US Metropolitan Areas using machine learning models on public tweets

Although the recent rise and uptake of COVID-19 vaccines in the United States has been encouraging, there continues to be significant vaccine hesitancy in various geographic and demographic clusters of the adult population. Surveys, such as the one conducted by Gallup over the past year, can be useful in determining vaccine hesitancy, but can be expensive to conduct and do not provide real-time data. At the same time, the advent of social media suggests that it may be possible to get vaccine hesitancy signals at an aggregate level, such as at the level of zip codes. Theoretically, machine learning models can be learned using socioeconomic (and other) features from publicly available sources. Experimentally, it remains an open question whether such an endeavor is feasible, and how it would compare to non-adaptive baselines. In this article, we present a proper methodology and experimental study for addressing this question. We use publicly available Twitter data collected over the previous year. Our goal is not to devise novel machine learning algorithms, but to rigorously evaluate and compare established models. Here we show that the best models significantly outperform non-learning baselines. They can also be set up using open-source tools and software.Author summary: The rapid development of COVID-19 vaccines has been touted as a miracle of modern medicine and industry-government partnerships. Unfortunately, vaccine hesitancy has been stubbornly high in many countries, including within the United States. Surveys, such as those conducted by organizations such as Gallup, have historically proved to be useful tools with which to poll a representative sample of people on vaccine sentiment, but can be expensive to administer and tend to rely on small sample sizes. As a complementary alternative, we propose to use publicly streaming data from Twitter to quantify vaccine hesitancy at the level of zip-codes in near real-time. Unfortunately, the noise and bias often present in social media begs the practical question of whether a cost-effective approach is feasible. In this article, we propose both a methodology, as well as an experimental study, for addressing these challenges using simple machine learning models. Our goal is not to devise novel algorithms but to rigorously evaluate and compare established models. Using public US-based Twitter data collected in the wake of the pandemic, we find that learning-based models outperform non learning-based models by significant margins. The methods we evaluate can easily be set up using open-source software packages.