Scheduling finite difference approximations for DAG-modeled large scale applications

Abstract

An increasing number of scientific domains are confronted with the arduous task of managing large scale applications. For such applications, gradient estimations come at a large computational cost. Despite notable advances in automatic differentiation during the last years, its use in this context may reveal too costly in memory, inadequate for parallel architecture or require expert knowledge. For these reasons, we investigate an alternative approach that uses the finite difference method to evaluate the gradient of functions modeled as a directed acyclic graph. This approach enables the reuse of partial results from previous partial derivatives evaluations and thus reduces the computational cost. We identify a discrete optimization problem arising in the limitedmemory context of large scale applications that aims to maximize the computational efficiency of the gradient approximation by scheduling the partial derivatives. This optimization problem is extended to consider the partitioning of the computations on multiple processors.We further derive some properties of these optimization problems, such as their upper bound on performance gains. Following a brief description of algorithms designed to obtain sensible solutions for both problems, we study the increase in performance resulting from sequential and parallel schedules obtained for synthetic DAGs. Finally, we employ this approach to accelerate the gradient evaluation of DAGs representing real evolutionary biology models. For one of these large scale applications, our approach is shown to be nearly 400 times faster than a state-of-The-Art software in sequential, and more than 11,000 times faster when using 256 processors.