Non busy-waiting Barrier in MPI

[j***@gmail.com]

Hi there,

I am writing an application where the master process needs to take a
long time to load some large files first. During this time, the other
processes have nothing to do but wait. So I have a MPI_Barrier() at
the end of the loading. What is happening in my runs on a shared
memory opteron Linux machine (four quad core chips) is that not only
the master process but some of the slave processes as well are taking
100% of CPU usage. I think this is caused by the busy waiting (spin
waiting, polling) of the Waiting functions of MPI. This feature may be
very efficient for most of MPI applications but wasted a lot of CPU
time for other users on the same machine. So I am wondering if there
is a way to let the slave nodes sleep while waiting.

I did some research and found that IBM MPI has the MP_WAIT_MODE and
the SUPER-UX MPI/XS library has MPISUSPEND to choose the waiting mode,
either polling or sleeping. I am using the MPICH2 library on a linux
platform. I am wondering if there is something I can use to achieve
the same purpose.

Thank you very much.
Shi

[Michael Hofmann]

Post by j***@gmail.com
I am writing an application where the master process needs to take a
long time to load some large files first. During this time, the other
processes have nothing to do but wait. So I have a MPI_Barrier() at
the end of the loading.

Is the barrier really necessary? If the other processes depend on the
master, they automatically wait for the master (without a barrier).
Barriers are only necessary to enforce determinism in non-deterministic
code.

Post by j***@gmail.com
What is happening in my runs on a shared
memory opteron Linux machine (four quad core chips) is that not only
the master process but some of the slave processes as well are taking
100% of CPU usage. I think this is caused by the busy waiting (spin
waiting, polling) of the Waiting functions of MPI. This feature may be
very efficient for most of MPI applications but wasted a lot of CPU
time for other users on the same machine. So I am wondering if there
is a way to let the slave nodes sleep while waiting.

No, MPI says nothing about busy waiting, sleeping etc. It's up to the MPI
implementation, how to wait at a barrier. However, you can try to use a
broadcast (with a zero length message) with the master process as root.
This can be used as a barrier, too (maybe without busy waiting).


Michael

[j***@gmail.com]

Thank you for your reply.

Post by Michael Hofmann
Is the barrier really necessary? If the other processes depend on the
master, they automatically wait for the master (without a barrier).
Barriers are only necessary to enforce determinism in non-deterministic
code.

I guess it is not really about Barrier. I removed Barrier and left it
with a BCast. The same thing happened.
I guess BCast calls the same busy waiting function as Barrier does.
Do you know any non-busy waiting function?
Thanks a lot.

Shi

Post by Michael Hofmann
Michael

[Georg Bisseling]

For MPICH2 it should be possible to configure
it to use the socket device exclusively. This
device can sleep() on incoming data be it receive,
bcast or barrier.

You will however get more latency compared to a
shared memory device.