Newtonian Methods

Abstract

The present chapter details the most important approach (by far) to compute a descent direction at each iteration of a minimization algorithm. This is the quasi-Newton method, defined in §4.4. To use another direction cannot be considered without a serious motivation; this has been true for decades and will probably remain so for several more years. 4.1 Preliminaries To solve the optimality condition g(x) = 0, we have seen in Chap. 2 essentially two possibilities for the direction: the gradient, or a vector of the canonical basis. Both are bad; to do better, let us recall the Newton principle. Starting from the current iterate x k , replace g by its linear approximation: g(x k + d) = g(x k) + g (x k)d + o(|d|) where g (x k) is the Jacobian of g at x k. Using the notation from §1.3, our problem (P) is to find d such that g(x k +d) = 0; to obtain the model-problem (P k), we then neglect the term o(|d|); this gives the linearized problem g(x k)+ g (x k)d = 0. Its solution is d N = −[g (x k)] −1 g(x k) (when g (x k) is invertible), and the next iterate is x N = x k + d N. In the case of an optimization problem, g is the gradient of f , g = f is its Hessian. Just as g was approximated to first order, f can be approximated to second order: f (x k + d) = f (x k) + (f (x k), d) + 1 2 (d, f (x k)d) + o(|d| 2). The quadratic approximation thus obtained is minimized (in the elliptic case) when its gradient vanishes: f (x k) + f (x k)d = 0. We realize an evidence: Newton's method on min f (x) is Newton's method on f (x) = 0. The big advantage of Newton's method is well known: it converges very fast. Theorem 4.1. If f is continuous and invertible near a solution x * , then the convergence of Newton's method is Q-superlinear. If, in addition, f ∈ C 3 , this convergence is Q-quadratic.