IDEAS home Printed from
   My bibliography  Save this paper

Optimizing Stata for Analysis of Large Data Sets


  • Joseph Canner

    () (Johns Hopkins University School of Medicine)

  • Eric Schneider

    (Johns Hopkins School of Medicine)


As with most programming languages, there can be multiple ways to do a task in Stata. Using modern CPUs with adequate memory, most Stata data processing commands run so quickly on small- or moderate-sized data sets that it is impossible to tell whether one command performs more efficiently than another. However, when analyzing large data sets such as the Nationwide Inpatient Sample (NIS), with about 8 million records per year (~3.5GB), this choice can make a substantial difference in performance. Using the Stata timer command, we performed standardized benchmarks of common programming tasks, such as searching the NIS for a list of ICD-9 codes, converting string data to numeric, and putting numeric variables in categories. For example, the inlist function can achieve significant performance gains compared to the equivalent “var==exp1 | var==exp2” notation (38% improvement) or using foreach loops (300% improvement). Using the real and subinstr functions to remove characters from strings and convert them to numbers is about 20 times faster than the destring command. The inlist, inrange, and recode functions also perform considerably better than the equivalent recode commands (13-70 times faster), especially for string variables, and are often easier to write and to read.

Suggested Citation

  • Joseph Canner & Eric Schneider, 2013. "Optimizing Stata for Analysis of Large Data Sets," 2013 Stata Conference 10, Stata Users Group.
  • Handle: RePEc:boc:norl13:10

    Download full text from publisher

    File URL:
    Download Restriction: no

    References listed on IDEAS

    1. Aaberge, Rolf & Dagsvik, John K & Strom, Steinar, 1995. " Labor Supply Responses and Welfare Effects of Tax Reforms," Scandinavian Journal of Economics, Wiley Blackwell, vol. 97(4), pages 635-659, December.
    2. Train,Kenneth E., 2009. "Discrete Choice Methods with Simulation," Cambridge Books, Cambridge University Press, number 9780521766555, December.
    3. Arne Risa Hole, 2007. "Fitting mixed logit models by using maximum simulated likelihood," Stata Journal, StataCorp LP, vol. 7(3), pages 388-401, September.
    4. Peter Haan & Arne Uhlendorff, 2006. "Estimation of multinomial logit models with unobserved heterogeneity using maximum simulated likelihood," Stata Journal, StataCorp LP, vol. 6(2), pages 229-245, June.
    5. Hoynes, Hilary Williamson, 1996. "Welfare Transfers in Two-Parent Families: Labor Supply and Welfare Participation under AFDC-UP," Econometrica, Econometric Society, vol. 64(2), pages 295-332, March.
    6. Arthur van Soest, 1995. "Structural Models of Family Labor Supply: A Discrete Choice Approach," Journal of Human Resources, University of Wisconsin Press, vol. 30(1), pages 63-88.
    7. Daniel McFadden & Kenneth Train, 2000. "Mixed MNL models for discrete response," Journal of Applied Econometrics, John Wiley & Sons, Ltd., vol. 15(5), pages 447-470.
    Full references (including those not matched with items on IDEAS)

    More about this item

    NEP fields

    This paper has been announced in the following NEP Reports:


    Access and download statistics


    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:boc:norl13:10. See general information about how to correct material in RePEc.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: (Christopher F Baum). General contact details of provider: .

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    We have no references for this item. You can help adding them by using this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service hosted by the Research Division of the Federal Reserve Bank of St. Louis . RePEc uses bibliographic data supplied by the respective publishers.