Calling Mpi Binary In Serial As Subprocess Of Mpi Application

I have a large parallel (using MPI) simulation application which produces large amounts of data. In order to evaluate this data I use a python script. What I now need to do is to r

Solution 1:

No, there is neither an elegant nor an official way to do this. The only officially supported way to execute other programs from within an MPI application is the use of MPI_Comm_spawn. Spawning child MPI processes via simple OS mechanisms like the one provided by subprocess is dangerous and could even have catastrophic consequences in certain cases.

While MPI_Comm_spawn does not provide a mechanism to find out when the child process has exited, you could kind of simulate it with an intercomm barrier. You will still face problems since the MPI_Comm_spawn call does not allow for the standard I/O to be redirected arbitrarily and instead it gets redirected to mpiexec/mpirun.

What you could do is to write a wrapper script that deletes all possible pathways that the MPI library might use in order to pass session information around. For Open MPI that would be any environment variable that starts with OMPI_. For Intel MPI that would be variables that start with I_. And so on. Some libraries might use files or shared memory blocks or some other OS mechanisms and you'll have to take care of those too. Once any possible mechanism to communicate MPI session information has been eradicated, you could simply start the executable and it should form a singleton MPI job (that is, behave as if run with mpiexec -n 1).