Quasi-Newton type proximal gradient method for nonconvex nonsmooth composite optimization problems

In this paper, we propose two quasi-Newton type proximal gradient methods for a class of nonconvex nonsmooth composite optimization problems, where the objective function is the sum of a smooth nonconvex function and a strictly increasing concave differentiable function composited with a convex nonsmooth function. The first proposed method is called quasi-Newton proximal gradient (QNPG) method, where the variable metric of the proximal operator adopts a quasi-Newton update strategy. The global convergence of QNPG is established under the Kurdyka-Łojasiewicz framework. However, proximal operators with quasi-Newton matrices are not easy to compute for some practical problems. Therefore we further give a general framework for proximal gradient method. Such a framework relies on an implementable inexactness condition for the computation of the proximal operator and on a line search procedure, where the line search directions can be selected arbitrarily. We prove that the line search criterion is well defined and the convergence of subsequences. Additionally, numerical simulations on an image processing model demonstrate the feasibility and effectiveness of the proposed methods.