Training robust neural network by importance sampling and minibatching

Mini-batch stochastic gradient descent serves as a prevalent technique, employing unbiased gradient estimation by uniformly sampling batches of instances, to train deep learning models. Despite sustained efforts to tackle large-scale optimization problems, a significant challenge remains: selecting training instances that enable convergence comparable to that achieved by training on the entire dataset, without compromising overall generalization ability. In this paper, we introduce a selection strategy based on importance sampling and minibatching, which calculates importance scores by simultaneously minimizing the variance of the gradients within evenly divided bins. The optimal sampling distribution is proportional to the upper bound of L2 norm of the minibatch gradients and can be efficiently computed at negligible cost. Extensive experiments conducted across multiple datasets and network architectures validate the exceptional performance of the proposed algorithm. The results demonstrate its superiority over various state-of-the-art strategies and its comparable performance with the full gradient estimation on the entire dataset.