Fast Multi-Step Optimization with Deep Learning for Data-Centric Supercomputing

Abstract

Following the improvement in data acquisition technology and in analysis capability, we develop an optimization method for data-centric supercomputing, which can utilize large-scale data. Prior information of the solution of these optimization problems is often available, and the use of such information is expected to improve the efficiency of the optimization process and the accuracy of its solution. Targeting use on supercomputing environments such as Big Data and Extreme-scale Computing (BDEC) resources, we develop a fast parameter optimization method that combines data-centric fast simulation methods and deep learning. This method comprises multiple steps, i.e., an approximate solution is obtained in a short time using prior information on many compute nodes, and the approximate solution is refined using a small number of compute nodes. Thus, optimum solutions can be obtained in a short time based on the required accuracy, even on many compute-node environments without fast interconnects. The developed method is expected to provide useful knowledge for related optimization problems computed on distributed ubiquitous computing resources using large observation data obtained by the Internet of Things (IoT) and fifth-generation (5G) networks.