Fitting generalized linear models when the data exceeds available memory

Despite the increase in random access memory (RAM) capacity and the decrease in RAM prices in the years since Stata was first released, the increase in the size of data sets in recent years can still exceed available RAM. This is particularly true for those who are using Stata on a personal laptop or desktop instead of an enterprise server. Accordingly, there is a need for statistical tools that can read small chunks of data from disk, perform calculations on those chunks, accumulate intermediate results, and produce final results that are the same as those obtained by performing the entire calculation in memory. The most ubiquitous statistical method is the generalized linear model (GLM), and mathematical methods have been available for many years to update the Q-R or Cholesky decomposition matrices with small chunks of data. Thomas Lumley’s R command bigglm uses Fortran functions published by Alan J. Miller in 1992 and freely available as Algorithm AS 274. We have developed –bigglm- for Stata using the same functions, as well as expanding the library of available family and link functions. The current version can read Stata datasets as well as import data from an ODBC source. In the presentation we will discuss the limitations of the current approach and suggest areas for improvement.