Backwards Differentiation in AD and Neural Nets: Past Links and New Opportunities

Abstract

Backwards calculation of derivatives - sometimes called the reverse mode, the full adjoint method, or backpropagation - has been developed and applied in many fields. This paper reviews several strands of history, advanced capabilities and types of application - particularly those which are crucial to the development of brain-like capabilities in intelligent control and artificial intelligence.