Mining Twitter data for fun and profit

Twitter feed data has increasingly become a rich source of information, both for commercial marketing purposes and for social science research. In the early days of Twitter, researchers could access the Twitter API with a simple URL. In order to control the size of data requests and to track who is accessing what data, Twitter has instituted some security policies that make this much more difficult for the average researcher. Users must obtain unique keys from Twitter, create a timestamp, generate a random string, combine this with the actual data request, hash this string using HMAC-SHA1, and submit all of this to the Twitter API in a timely fashion. Large requests must be divided into multiple smaller requests and spread out over a period of time. Our particular need was to obtain Twitter user profile information for over 800 users who had previously tweeted about surgery. We developed a Stata command to automate this process. It includes a number of bitwise operator functions and other tools that could be useful in other applications. We will discuss the unique features of this command, the tools required to implement it, and the feasibility of extending this example to other data request types.