Data Nuggets in Supervised Learning

Big data presents many challenges in modern statistics and data analysis. While a large number of observations can lead to increased precision in statistical parameter estimation and prediction, computational and storage costs may present a problem. Since there is often significant redundancy in large data in a lower dimensional setting, it seems reasonable that big datasets can be compressed to a smaller number of observations with comparable statistical performance, where the amount of compression scales with the dimensionality. We propose an extension of the “data nuggets” methodology of Beavers et al. (2020) for a compression-based approach to statistical modeling in big data. We utilize the linear regression model to showcase the idea, establish a theoretical foundation, and explore finite-sample performance via simulation analysis. Data nuggets are shown to provide a significant improvement over random sampling in model parameter estimation and out-of-sample prediction performance in the linear regression setting, and the concept is promising for other models as well.