Data cleaning in Stata using Internet search engines
AbstractOpen-ended questions can be a nightmare for statistical processing. Any mistake in spelling can result in a mismatch during merging, or multiple counting of the same object. For example, the answers to the "place-of-birth" question might be "Chicago" and "San Francisco", but in practice they are often "Chicaga" and "SanFrancisko". Manual correction of hundreds of answers is tedious, and becomes infeasible with a larger dataset. For a long time, algorithms like SOUNDEX remained the only alternative for researchers. A new Stata command allows taking advantage of Internet search engines, like Google or Yahoo to find proper substitutes for an unclear word or multiple words. The distinctive feature of the search engines is that they rely not only on the spelling similarity, but are also context driven: other words may affect the suggestion, such as including "city" into the query. This will hint to the search engine to give more priority to the names of cities. This presentation will demonstrate this new command and explain the main steps necessary to programmatically acquire information available on the Internet and convert it into Stata-usable format.
Download InfoIf you experience problems downloading a file, check if you have the proper application to view it first. In case of further problems read the IDEAS help page. Note that these files are not on the IDEAS site. Please be patient as the files may be large.
Bibliographic InfoPaper provided by Stata Users Group in its series Canadian Stata Users' Group Meetings 2009 with number 08.
Date of creation: 06 Nov 2009
Date of revision:
data cleaning; search engine; spelling correction;
This paper has been announced in the following NEP Reports:
- NEP-ALL-2009-11-21 (All new papers)
You can help add them by filling out this form.
reading list or among the top items on IDEAS.Access and download statistics
For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: (Christopher F Baum).
If references are entirely missing, you can add them using this form.