Excluding procs from MPI_COMM_WORLD

[Chris]

Hi folks,

As a starter I've got a little MPI problem, which I'm sure is easy to
solve for you.

Imagine a parallel program which runs for 4 processes maximum for a
given problem. It uses broadcast to communicate some data. If I now
start the program with, say 32, processes, the broadcast primitive
include all 28 not necessary processes as well. Is there any chance to
exclude those 28 procs from MPI_COMM_WORLD, or do I need to create a
new communication handle for the 4 processes?

I tried to let the 28 procs call MPI_Finalize(), but this only stucked
because apparently MPI_Bcast still wants to include them and waits for
them calling MPI_Bcast as well.

Thanks for your help in advance
/Chris

[Christian]

Hi Chris,

Post by Chris
Imagine a parallel program which runs for 4 processes maximum for a
given problem. It uses broadcast to communicate some data. If I now
start the program with, say 32, processes, the broadcast primitive
include all 28 not necessary processes as well. Is there any chance to
exclude those 28 procs from MPI_COMM_WORLD, or do I need to create a
new communication handle for the 4 processes?
I tried to let the 28 procs call MPI_Finalize(), but this only stucked
because apparently MPI_Bcast still wants to include them and waits for
them calling MPI_Bcast as well.

MPI_Bcast() is defined to be a collective operation, meaning that
every process within the specified communicator is involved (i.e. your
last approach won't work). The MPI standard does not define something
like MPI_Mcast() where you can selectively send something to an
arbitrary set of processes. Therefore you have to create a new
communicator. One possible way to achieve this is to utilize the
MPI_Comm_split() function, e.g.:

int rank, color;
MPI_Comm_rank(MPI_COMM_WORLD, &rank);
color = (rank < 4);
MPI_Comm_split(MPI_COMM_WORLD, color, 0, &newcomm);

This will create two different communicators (assuming more than 4 MPI
processes in MPI_COMM_WORLD):
- one communicator holding all MPI processes with rank < 4 (i.e. 0, 1,
2, 3) to do your actual work
- and the remaining MPI processes (with rank >= 4)

Good luck!

Christian

[Chris]

Hi Christian,

Post by Christian
int rank, color;
MPI_Comm_rank(MPI_COMM_WORLD, &rank);
color = (rank < 4);
MPI_Comm_split(MPI_COMM_WORLD, color, 0, &newcomm);

Thanks, that worked out perfectly!

Post by Christian
This will create two different communicators (assuming more than 4 MPI

I read the same in the MPI documentation, but do not understand one
detail: What is the second communicator then? One is newcomm, is the
rest now in MPI_COMM_WORLD?! Would be great if you could clarify this
to me.

Cheers,
/Chris

[Christian]

Hi Chris,

Post by Chris

Post by Christian
This will create two different communicators (assuming more than 4 MPI

I read the same in the MPI documentation, but do not understand one
detail: What is the second communicator then? One is newcomm, is
the rest now in MPI_COMM_WORLD?! Would be great if you could
clarify this to me.

No, MPI_COMM_WORLD is a pre-defined communicator that contains (in the
static-process-model) all MPI processes. The collective operation
MPI_Comm_split() creates a single new communicator (i.e. local handle)
for every (because we use MPI_COMM_WORLD) MPI process. The 'color'
argument describes which of those "local handles" are "grouped"
together (see chapter 5.3. "Group Management" in the MPI standard).
In your example we would create 32 new local communicator handles (one
for each MPI process) which are then grouped into two disjoint sets
containing 4 and 28 MPI processes.

Christian

[Chris]

Post by Christian
In your example we would create 32 new local communicator handles (one
for each MPI process) which are then grouped into two disjoint sets
containing 4 and 28 MPI processes.

Got it. Muchas gracias! :)

[l***@yahoo.com]

Post by Christian
In your example we would create 32 new local communicator handles (one
for each MPI process) which are then grouped into two disjoint sets
containing 4 and 28 MPI processes.

So when you want to broadcast to the 4 processors, you use newcomm,
right? What if you now want to broadcast to the other 28 processors?
Do you first have to create a new group/communicator for them first?

[Chris]

Post by l***@yahoo.com
So when you want to broadcast to the 4 processors, you use newcomm,
right? What if you now want to broadcast to the other 28 processors?
Do you first have to create a new group/communicator for them first?

Exactly. Like this I suppose:

MPI_Comm_split(MPI_COMM_WORLD, (id < 4), 0, &newcomm4procs);
MPI_Comm_split(MPI_COMM_WORLD, (id >= 4), 0, &newcomm28procs);

[David Cronk]

Post by l***@yahoo.com

Post by Christian
In your example we would create 32 new local communicator handles (one
for each MPI process) which are then grouped into two disjoint sets
containing 4 and 28 MPI processes.

So when you want to broadcast to the 4 processors, you use newcomm,
right? What if you now want to broadcast to the other 28 processors?
Do you first have to create a new group/communicator for them first?

It depends on the root. When you say broadcast, what process is doing
the broadcast? If one of the 4 processes is broadcasting to the other
3, then use newcom. If one of the 28 processes is broadcasting to the
other 27 in that group, then use newcom. If one of the 4 processes is
broadcasting to the other 28 processes, you will need to create an
intercommunicator (see MPI_INTERCOMM_CREATE).

Hope this helps.

Dave.

[l***@yahoo.com]

Post by David Cronk

Post by l***@yahoo.com

Post by Christian
In your example we would create 32 new local communicator handles (one
for each MPI process) which are then grouped into two disjoint sets
containing 4 and 28 MPI processes.

So when you want to broadcast to the 4 processors, you use newcomm,
right? What if you now want to broadcast to the other 28 processors?
Do you first have to create a new group/communicator for them first?

It depends on the root. When you say broadcast, what process is doing
the broadcast? If one of the 4 processes is broadcasting to the other
3, then use newcom. If one of the 28 processes is broadcasting to the
other 27 in that group, then use newcom. If one of the 4 processes is
broadcasting to the other 28 processes, you will need to create an
intercommunicator (see MPI_INTERCOMM_CREATE).
Hope this helps.
Dave.

Yeah, I'm talking about one of the 28 processes broadcasting to the
other 27 in that group. So just have the 28 processors call
MPI_Bcast?

By splitting the 32 processors into 2 groups, we then have both groups
called newcomm, and to differentiate which group is involved in the
broadcast, just have the relevant processors call the routine?

[David Cronk]

Post by l***@yahoo.com

Post by David Cronk

Post by l***@yahoo.com

Post by Christian
In your example we would create 32 new local communicator handles (one
for each MPI process) which are then grouped into two disjoint sets
containing 4 and 28 MPI processes.

So when you want to broadcast to the 4 processors, you use newcomm,
right? What if you now want to broadcast to the other 28 processors?
Do you first have to create a new group/communicator for them first?

It depends on the root. When you say broadcast, what process is doing
the broadcast? If one of the 4 processes is broadcasting to the other
3, then use newcom. If one of the 28 processes is broadcasting to the
other 27 in that group, then use newcom. If one of the 4 processes is
broadcasting to the other 28 processes, you will need to create an
intercommunicator (see MPI_INTERCOMM_CREATE).
Hope this helps.
Dave.

Yeah, I'm talking about one of the 28 processes broadcasting to the
other 27 in that group. So just have the 28 processors call
MPI_Bcast?

Yeah. Just use newcomm.

Post by l***@yahoo.com
By splitting the 32 processors into 2 groups, we then have both groups
called newcomm, and to differentiate which group is involved in the
broadcast, just have the relevant processors call the routine?

Sort of. Both groups are referenced by the communicator newcomm. (MPI
has both groups and communicators, though they conceptually can often be
thought of as the same. I could explain why the MPI forum did this, but
it may be even more confusing.) Communicators are local. newcomm in
one of the processes from the group of 4 refers just to those 4
processes. newcomm in one of the processes from the group of 28 refers
just to those 28 processes.

As I said, this is a simplification, but for most purposes this
abstraction will work fine.

Dave.

[l***@yahoo.com]

Post by David Cronk

Post by l***@yahoo.com

Post by David Cronk

Post by l***@yahoo.com

Post by Christian
In your example we would create 32 new local communicator handles (one
for each MPI process) which are then grouped into two disjoint sets
containing 4 and 28 MPI processes.

So when you want to broadcast to the 4 processors, you use newcomm,
right? What if you now want to broadcast to the other 28 processors?
Do you first have to create a new group/communicator for them first?

It depends on the root. When you say broadcast, what process is doing
the broadcast? If one of the 4 processes is broadcasting to the other
3, then use newcom. If one of the 28 processes is broadcasting to the
other 27 in that group, then use newcom. If one of the 4 processes is
broadcasting to the other 28 processes, you will need to create an
intercommunicator (see MPI_INTERCOMM_CREATE).
Hope this helps.
Dave.

Yeah, I'm talking about one of the 28 processes broadcasting to the
other 27 in that group. So just have the 28 processors call
MPI_Bcast?

Yeah. Just use newcomm.

Post by l***@yahoo.com
By splitting the 32 processors into 2 groups, we then have both groups
called newcomm, and to differentiate which group is involved in the
broadcast, just have the relevant processors call the routine?

Sort of. Both groups are referenced by the communicator newcomm. (MPI
has both groups and communicators, though they conceptually can often be
thought of as the same. I could explain why the MPI forum did this, but
it may be even more confusing.) Communicators are local. newcomm in
one of the processes from the group of 4 refers just to those 4
processes. newcomm in one of the processes from the group of 28 refers
just to those 28 processes.
As I said, this is a simplification, but for most purposes this
abstraction will work fine.
Dave.
--
Dr. David Cronk, Ph.D. phone: [phone]
Research Director fax: [phone]
Innovative Computing Lab http://www.cs.utk.edu/~cronk
University of Tennessee, Knoxville

Thank you, I get it now!