batchtools: Tools for R to work on batch systems

Abstract

The R (R Core Team 2016) package batchtools is the successor of the BatchJobs package (Bischl et al. 2015). It provides an implementation of a Map-like operation to define and asynchronously execute jobs on a variety of parallel backends: • Local (blocking) execution in the current R session or in an externally spawned R process (intended for debugging and prototyping) • Local (non-blocking) parallel execution using parallel's multicore backend (R Core Team 2016) or snow's socket mode (Tierney et al. 2016). • Execution on loosely connected machines using SSH (including basic resource usage control). • Docker Swarm • IBM Spectrum LSF • OpenLava • Univa Grid Engine (formerly Oracle Grind Engine and Sun Grid Engine) • Slurm Workload Manager • TORQUE/PBS Resource Manager Extensibility and user customization are important features as configuration on high-performance computing clusters is often heavily tailored towards very specific require-ments or special hardware. Hence, the interaction with the schedulers uses a template engine for improved flexibility. Furthermore, custom functions can be hooked into the package to be called at certain events. As a last resort, many utility functions simplify the implementation of a custom cluster backend from scratch.