Data cleaning in Stata using Internet search engines

Open-ended questions can be a nightmare for statistical processing. Any mistake in spelling can result in a mismatch during merging, or multiple counting of the same object. For example, the answers to the "place-of-birth" question might be "Chicago" and "San Francisco", but in practice they are often "Chicaga" and "SanFrancisko". Manual correction of hundreds of answers is tedious, and becomes infeasible with a larger dataset. For a long time, algorithms like SOUNDEX remained the only alternative for researchers. A new Stata command allows taking advantage of Internet search engines, like Google or Yahoo to find proper substitutes for an unclear word or multiple words. The distinctive feature of the search engines is that they rely not only on the spelling similarity, but are also context driven: other words may affect the suggestion, such as including "city" into the query. This will hint to the search engine to give more priority to the names of cities. This presentation will demonstrate this new command and explain the main steps necessary to programmatically acquire information available on the Internet and convert it into Stata-usable format.