A model aggregation approach for high-dimensional large-scale optimization

Bayesian optimization (BO) has been widely used in machine learning and simulation optimization. With the increase in computational resources and storage capacities, high-dimensional and large-scale optimization problems are becoming increasingly common. Empirically, tasks that seem high-dimensional in machine learning, such as hyper-parameter optimization, often reveal a markedly reduced intrinsic dimensionality. In this paper, we propose a Model Aggregation Method for Bayesian Optimization (MamBO) algorithm to efficiently solve high-dimensional large-scale optimization problems with low effective dimensionality. MamBO addresses the high dimensional and large-scale data set challenges simultaneously with a combination of data subsampling and subspace embeddings. At the same time, a model aggregation method is employed to mitigate the surrogate model uncertainty issue which is largely ignored in the embedding literature and practice. Our proposed model aggregation method reduces this lower-dimensional surrogate model uncertainty and improves the robustness of the BO algorithm. We derive an asymptotic bound for the proposed aggregated surrogate model and prove the convergence of the MamBO algorithm. Several experiments indicate that our algorithm achieves superior or comparable performance to state-of-the-art high-dimensional BO algorithms and is computationally faster.