Learning with Stochastic Gradient

Statistical learning often involves minimizing an objective function to find a suitable value of the model parameter. A simple and ubiquitous method for minimizing a differentiable objective function is Gradient Descent, which uses iterative parameter updates in the direction of steepest descent to minimize the objective. However, there are situations where the calculation of the objective function gradient is either analytically intractable or computationally infeasible. Two important examples are parameter estimation for Gibbs models and weight optimization in deep neural networks. Stochastic Gradient methods, which use a random but unbiased estimate of the full gradient, can be a useful tool for overcoming situations where the full gradient is unavailable. In the first half of the chapter, several theorems concerning the approximation of the true gradient from per-observation gradients are presented, and an important connection between Stochastic Gradient and Langevin Dynamics is discussed. The second section covers parameter estimation for Markov Random Field models, and the final section presents MCMC learning methods for deep image models. Stochastic gradient