Optimality conditions for MCMC in rare event risk analysis

Monte Carlo method is an indispensable tool in modern computational risk analysis for its robustness with complex systems that are increasingly ‘black-box’ in nature. Decades of research reveal that the ability to estimate small probabilities efficiently is intimately related to that of generating rare event samples. Subset Simulation offers a simple approach by sequentially propagating populations of samples failing increasing response thresholds. Markov Chain Monte Carlo (MCMC) machine-learns about rare events while maintaining the correct distribution for statistical estimation. Existing research is mostly focused on new algorithms or tuning hyperparameters through heuristics or empirical studies. Beyond conventional objectives, this work presents a general theory that establishes the conditions for an optimal MCMC algorithm in terms of minimizing the correlation between successive samples. It resolves the correlation into ‘failure mixing rate’, a new measure conducive to further analysis, for which the first two derivatives with respect to hyperparameters have been obtained analytically in terms of response gradient. The theory is illustrated with conditional sampling scheme for problems with linear and nonlinear response functions, high dimensions, and multiple failure modes. This work provides a pathway for optimizing MCMC for rare events in risk analysis. A number of questions of theoretical and computational nature are outstanding, calling for future research efforts.