On exhaustive sufficient dimension reduction methods for massive datasets

We introduce BIG-SAVE and BIG-DR, novel sufficient dimension reduction methods specifically designed to handle massive datasets, that challenge conventional computation. Our approaches extend traditional sufficient dimension reduction by adopting a divide-and-conquer strategy: the full dataset is partitioned into manageable chunks, sufficient dimension reduction is performed independently on each subset, and the resulting subspaces are integrated using a proximity-based aggregation method. Unlike existing sufficient dimension reduction techniques, BIG-SAVE and BIG-DR are optimized for parallel computing and memory mapping, allowing them to process data that exceed the limits of available random access memory. Moreover, our approach ensures robust estimation stability across chunks and maintains exhaustiveness, ensuring full recovery of the central subspace under mild conditions. We demonstrate the practical performance of our proposed methods through extensive simulation studies and real-world data applications. Our results show that BIG-SAVE and BIG-DR achieve competitive estimation accuracy, significant computational efficiency, and scalability making them suitable for modern large-scale data analysis tasks in scientific and industrial domains.