Sound and efficient dynamic verification of MPI programs with probe non-determinism

Abstract

We consider the problem of verifying MPI programs that use MPI-Probe and MPI-Iprobe. Conventional testing tools, known to be inadequate in general, are even more so for testing MPI programs containing MPI probes. A few reasons are: (i) use of the MPI-ANY-SOURCE argument can make MPI probes non-deterministic, allowing them to match multiple senders, (ii) an MPI-Recv that follows an MPI probe need not match the MPI-Send that was successfully probed, and (iii) simply re-running the MPI program, even with schedule perturbations, is insufficient to bring out all behaviors of an MPI program using probes. We develop several key insights that help develop an elegant solution: prioritizing MPI processes during dynamic verification, handling non-determinism, and safe handling of probe loops. These solutions are incorporated into a new version of our dynamic verification tool ISP. ISP is now able to efficiently and soundly verify larger MPI examples, including MPI-BLAST and ADLB. © 2009 Springer Berlin Heidelberg.