On automatic differentiation and algorithmic linearization

Abstract

We review the methods and applications of automatic differentiation, a research and development activity, which has evolved in various computational fields since the mid 1950’s. Starting from very simple basic principles that are familiar from school, one arrives at various theoretical and practical challenges. The resulting activity encompassesmathematical research and software development; it is now often referred to as algorithmic differentiation. From a geometrical and algebraic point of view, differentiation amounts to linearization, a concept that naturally extends to infinite dimensional spaces. In contract to other surveys, we will emphasize this interpretation as it has become more important recently and also facilitates the treatment of nonsmooth problems by piecewise linearization.