Standard Methods For Constrained Optimization

Consider the general constrained optimization problem: $$\begin{aligned} {\mathop {{{\mathrm{minimize\,}}}}_\mathbf{x}}&f(\mathbf{x})\nonumber \\ \text {such that }&g_j(\mathbf{x})\le 0\ \ j=1,2,\dots , m\\&h_j(\mathbf{x})=0\ \ j=1,2,\dots , r.\nonumber \end{aligned}$$ The most simple and straightforward approach to handling constrained problems of the above form is to apply a suitable unconstrained optimization algorithm to a penalty function formulation of constrained problem. Unfortunately the penalty function method becomes unstable and inefficient for very large penalty parameter values if high accuracy is required. A remedy to this situation is to apply the penalty function method to a sequence of sub-problems, starting with moderate penalty parameter values, and successively increasing their values for the sub-problems. Alternatively, the Lagrangian function with associated necessary Karush-Kuhn-Tucker (KKT) conditions and duality serve to solve constrained problems that has led to the development of the Sequential Quadratic Programming (SQP) method that applies $$\text {Newton's}$$ method to solve the KKT conditions.