Quasi-Newton Methods

The idea of these methods is not to use the Hessian ∇2f(xk) of the minimizing function in the current point at every iteration, but instead to use an approximation of it. In this chapter, consider Bk as an approximation to ∇2f(xk) and Hk an approximation to the inverse Hessian ∇2f(xk)−1. Obviously, both these approximations Bk or Hk may be used, but the use of Bk involves the solving of a linear algebraic system. The quasi-Newton method requires only the gradient of the minimizing function, gradient which must be supplied by the user at every iteration. The purpose of this chapter is to present these methods together with the theoretical aspects concerning their convergence to solution, as well as their performances for solving large-scale complex optimization problems and applications. In quasi-Newton methods, the approximations to the Hessian (or to the inverse Hessian) may be achieved through matrices of rank one or through matrices of rank two. From the multitude of quasi-Newton methods, this chapter will present only the rank-two updating methods, that is, Davidon-Fletcher-Powell (DFP) and Broyden-Fletcher-Goldfarb-Shanno (BFGS), and the rank one, that is, Symmetric Rank 1 (SR1), together with some modifications of them.