Efficient, scalable emulation of stochastic simulators: A mixture density network based surrogate modeling framework

This work focuses on the task of emulating stochastic simulators that generate random results given consistent inputs. Existing stochastic surrogate models can have limitations in characterizing complex output distributions or suffer from data inefficiency and high-dimensionality issues. As a remedy, we propose to introduce Mixture Density Networks (MDNs) as an advanced stochastic surrogate model. Flexible neural networks are utilized to parametrize Gaussian mixture models (GMMs) to capture complex input-dependent output conditional densities. MDNs, like typical neural networks, use multiple layers of nonlinear transformations to effectively manage the potentially high-dimensional or large-volume inputs. Statistically, MDNs eliminate the need for random seed control or replication by directly learning to maximize the training data likelihood. To further enhance MDN model robustness and mitigate over-fitting, we incorporate the hard parameter sharing and variance regularization technique, and Bayesian optimization to search for optimal MDN hyperparameters, unlocking its full potential as stochastic surrogates. The efficacy of our MDN stochastic surrogate model is illustrated through various academic and realistic examples. MDN is demonstrated to be consistently more flexible, accurate, and data-efficient than current stochastic surrogate workhorses.