floydfan
2008-06-30 10:35:45 UTC
Hi,
I'm working at KISTI Supercomputing Center, Korea, as a member of
parallel computing support group.
Recently, we made a new supercom system (SUN x6420 * 192 nodes) with
Infiniband network connection among nodes and installed various mpi
libraries : MVAPICH, MVAPICH2 and OpenMPI.
While supporting application scientists, I found that, some codes work
well with OpenMPI, though they don't work with MVAPICH. Particularly,
these codes even work with MVAPICH, when I use several processors in
the same node; but they don't work if more than 2 nodes cooperate.
(i.e., when I use 16 processors in the same node, it works well... If
I use 16 processors in 2 nodes, it pends in the MPI communication
routine...)
My curiosity made me to test very simple code (which consists of
several MPI routines with small data size...) to find whether the
installation procedure was wrong... But, in a simple code, there's no
trouble in inter-node communication. On the other hand, some codes
that spend vast size memory show this kind of situation even though
they transfer only 4 bytes data (one integer).
As an application scientist myself, I can never understand this kind
of situation. So, I'd like to know the reason and when this situation
would happen. Why do some codes pend in MPI routines if I use multiple
nodes? And, is there any remedy? Also, what characteristics of the
code brings about this problem?
Please let me know the reason for this happening.
Thank you in advance.
Jeff
p.s. What is main difference between MVAPICH and OpenMPI?
I'm working at KISTI Supercomputing Center, Korea, as a member of
parallel computing support group.
Recently, we made a new supercom system (SUN x6420 * 192 nodes) with
Infiniband network connection among nodes and installed various mpi
libraries : MVAPICH, MVAPICH2 and OpenMPI.
While supporting application scientists, I found that, some codes work
well with OpenMPI, though they don't work with MVAPICH. Particularly,
these codes even work with MVAPICH, when I use several processors in
the same node; but they don't work if more than 2 nodes cooperate.
(i.e., when I use 16 processors in the same node, it works well... If
I use 16 processors in 2 nodes, it pends in the MPI communication
routine...)
My curiosity made me to test very simple code (which consists of
several MPI routines with small data size...) to find whether the
installation procedure was wrong... But, in a simple code, there's no
trouble in inter-node communication. On the other hand, some codes
that spend vast size memory show this kind of situation even though
they transfer only 4 bytes data (one integer).
As an application scientist myself, I can never understand this kind
of situation. So, I'd like to know the reason and when this situation
would happen. Why do some codes pend in MPI routines if I use multiple
nodes? And, is there any remedy? Also, what characteristics of the
code brings about this problem?
Please let me know the reason for this happening.
Thank you in advance.
Jeff
p.s. What is main difference between MVAPICH and OpenMPI?