Optimizing Shell Scripting Languages

Abstract

We present an optimizing compiler that dramatically improves the performance of shell script languages. Our prototype system, called Shark, performs a number of domainspecific optimizations. We show that, subject to some constraints, we can treat file I/O in shell languages like variables in regular programming languages. This insight allows us to leverage traditional compiler analyses to eliminate temporary files, transform file-based I/O to pipes, and extract coarse-grained parallelism. We show how we can further improve performance by eliminating process creation overhead, which can dominate the runtime of shell scripts. By modifying just a few lines of code, we can convert external commands like grep into dynamic libraries. Shark loads these libraries just once so that subsequent invocations execute with the speed of function calls, and then unloads them when they are no longer needed. We present results with our prototype demonstrating substantial speedups over the bash shell.