hp mpi with 32 processors not working

[golla nageswararao]

Hi all,
I installed HP MPI for linux and could ran my fortran code very
successfully with 32 processors. But the problem is the next day when
I tried to run the code the following came with 2 processors.
Process Information:

Node # 0 (pid= 11457) is active.
Node # 1 (pid= 11458) is active.

Model Input Parameters: ROMS/TOMS version 2.2
Monday - May 12, 2008 - 1:02:41 PM

-----------------------------------------------------------------------------
forrtl: severe (408): fort: (3): Subscript #1 of the array CVAL has
value -1789532123 which is less than the lower bound of 1

Image PC Routine Line
Source
myindia_2proc 0000000000A4E74E Unknown Unknown
Unknown
myindia_2proc 0000000000A4D94E Unknown Unknown
Unknown
myindia_2proc 0000000000A0970E Unknown Unknown
Unknown
myindia_2proc 00000000009D5DD5 Unknown Unknown
Unknown
myindia_2proc 00000000009D5034 Unknown Unknown
Unknown
myindia_2proc 0000000000837DAC Unknown Unknown
Unknown
myindia_2proc 0000000000810918 Unknown Unknown
Unknown
myindia_2proc 0000000000809C19 Unknown Unknown
Unknown
myindia_2proc 0000000000404679 Unknown Unknown
Unknown
myindia_2proc 000000000040450C Unknown Unknown
Unknown
myindia_2proc 00000000004043EE Unknown Unknown
Unknown
libc.so.6 0000002A95A701AE Unknown Unknown
Unknown
myindia_2proc 000000000040432A Unknown Unknown
Unknown
MPI Application rank 0 exited before MPI_Finalize() with status 152
forrtl: error (78): process killed (SIGTERM)
Image PC Routine Line
Source
libc.so.6 0000002A95A82720 Unknown Unknown
Unknown
libmpi.so.1 0000002A956E7FBA Unknown Unknown
Unknown

With 32 processors also it is not working. I am not understanding why
it is not working one day which has worked very fine. My system
configuration is RHEL-3.0, AMD opteron 64-bit processors. I had seen
whether it is memory problem, but all are unlimited when I typed
ulimit -a
Can anybody help me in this aspect.

Thanks in advance.

Wtih best regards,
G.NageswaraRao.

[Michael Hofmann]

Post by golla nageswararao
-----------------------------------------------------------------------------
forrtl: severe (408): fort: (3): Subscript #1 of the array CVAL has
value -1789532123 which is less than the lower bound of 1

I guess, this error message says that your program tries to access an element of array "CVAL" using an invalid subscript (-1789532123). This looks like an uninitialized variable.

Post by golla nageswararao
MPI Application rank 0 exited before MPI_Finalize() with status 152
forrtl: error (78): process killed (SIGTERM)

The process with rank 0 causes the error.

Post by golla nageswararao
Can anybody help me in this aspect.

I cannot say why it is working one day and failing the other day. There may be a million reasons. Nevertheless, the error messages give a clear advice where to start debugging.


Michael