Performance of user data collections uploads to HPCaaS infrastructure

Abstract

When offering the HPC as a Service middleware to remote users wishing them to use a HPC infrastructure, we are facing a problem of uploading large data collections for processing. On one hand, as users perform calculations on local infrastructures as well, they typically make use of the most common 1 Gb/s connectivity and the institution’s connectivity to the Internet is not influencing transfer rates. On the other hand, the institution offering the HPC as a Service middleware may be distant both geographically and network-wise. Furthermore, the amount of data to be transferred ranges from hundreds of gigabytes to several terabytes. In this paper, we study the common protocols used for secure data transfers and evaluate results on IT4Innovations infrastructure. Changes to the upload to HEAppE middleware developed in our institution are also discussed.