Boost with MPI

Am 10.06.2009, 01:17 Uhr, schrieb Philipp Kraus

Post by Philipp Kraus
Hi,
I'm new at MPI. I try to link my C++ programm against the MPI libs, but
--------
mpi-test.cpp:(.text+0x76): undefined reference to
`boost::mpi::environment::environment(int&, char**&, bool)'
mpi-test.cpp:(.text+0x7f): undefined reference to
`boost::mpi::communicator::communicator()'
mpi-test.cpp:(.text+0x88): undefined reference to
`boost::mpi::communicator::size() const'
mpi-test.cpp:(.text+0x94): undefined reference to
`boost::mpi::communicator::rank() const'
mpi-test.cpp:(.text+0x124): undefined reference to
`boost::mpi::environment::~environment()'
mpi-test.cpp:(.text+0x149): undefined reference to
`boost::mpi::environment::~environment()'
collect2: ld returned 1 exit status
--------
I run "mpic++ -o mpi-test mpi-test.cpp" with the first Boost example on
http://www.boost.org/doc/libs/1_39_0/doc/html/mpi/tutorial.html#mpi.c_mapping

Seems like the linker does not look for the Boost-MPI library. You need to
specify them separately:

"mpic++ -o mpi-test mpi-test.cpp
/opt/library/boost/1.39.0/lib/libboost_mpi-mt.a"

(Make sure that the path and the name of the libboost_mpi* file is
correct!)

If this works, you can fix your compiler environment by adding the
following to your /etc/profile file:

"export LIBRARY_PATH=$BOOST/lib:$LIBRARY_PATH"

Then you can compile Boost MPI programs the following way:

"mpic++ -o mpi-test mpi-test.cpp -lboost_mpi-mt"

(Again, the name "boost_mpi-mt" needs to correspond to the Boost-MPI
library file that you have installed in /opt/library/boost/1.39.0/..."!)

Michael

Philipp Kraus

2009-06-10 11:01:33 UTC

On 2009-06-10 11:34:50 +0200, "Michael Hofmann"

Seems

Post by Michael Hofmann
like the linker does not look for the Boost-MPI library. You need to
"mpic++ -o mpi-test mpi-test.cpp
/opt/library/boost/1.39.0/lib/libboost_mpi-mt.a"

okay, thanks, that works. But the *.a file is the static or dynamic
lib? static?
How can I link it against the dynamic? *.so?

Post by Michael Hofmann
(Make sure that the path and the name of the libboost_mpi* file is correct!)
If this works, you can fix your compiler environment by adding the
"export LIBRARY_PATH=$BOOST/lib:$LIBRARY_PATH"

No, in our systemconfig /etc/profile we use also LD_LIBRARY_PATH

Post by Michael Hofmann
"mpic++ -o mpi-test mpi-test.cpp -lboost_mpi-mt"
(Again, the name "boost_mpi-mt" needs to correspond to the Boost-MPI
library file that you have installed in /opt/library/boost/1.39.0/..."!)

I change the -lboost_mpi_mt to -libboost_mpi-gcc42-mt.a but it doesn't
work. It seem the LD_FLAGS arn't be set, is that right?
With env variable is this only für MPI? MPI_LDFLAGS?

Thanks a lot

Phil

Michael Hofmann

2009-06-10 12:33:17 UTC

Am 10.06.2009, 13:01 Uhr, schrieb Philipp Kraus

okay, thanks, that works. But the *.a file is the static or dynamic lib?
static?
How can I link it against the dynamic? *.so?

*.a is the static library (an archive with lots of *.o files inside). *.so
is the shared object file for dynamic linking.

You can also link against the *.so with:

"mpic++ -o mpi-test mpi-test.cpp
/opt/library/boost/1.39.0/lib/libboost_mpi*.so"

Post by Michael Hofmann
"export LIBRARY_PATH=$BOOST/lib:$LIBRARY_PATH"

No, in our systemconfig /etc/profile we use also LD_LIBRARY_PATH

LD_LIBRARY_PATH is used by the "loader" to search for *.so files when then
program is executed.

LIBRARY_PATH is used by the compiler/linker to search for library files
(both *.a and *.so). If LIBRARY_PATH is set correctly, you can use the
(short) option "-lboost_mpi-gcc42-mt" to link against your Boost MPI
library (dynamic linking unless otherwise specified, e.g. with "-static").
In this case, the linker finds the library file by using LIBRARY_PATH and
the name used in the "-l<name>" option.

As an alternative, you can also specify the full path to the library file
manually (/opt/library/boost/1.39.0/lib/libboost_mpi...), e.g. to test
whether linking to this file is possible or not.

I change the -lboost_mpi_mt to -libboost_mpi-gcc42-mt.a but it doesn't
work. It seem the LD_FLAGS arn't be set, is that right?

You need to do two things:
1. set LIBRARY_PATH to /opt/library/boost/1.39.0/lib
2. use the option "-lboost_mpi-gcc42-mt"
^^^^^^^^^^^^^^^^^^
The full path, the prefix "lib" and the suffix ".a" or ".so" is added
automatically!

Michael

Philipp Kraus

2009-06-10 12:47:59 UTC

On 2009-06-10 14:33:17 +0200, "Michael Hofmann"

Post by Philipp Kraus
okay, thanks, that works. But the *.a file is the static or dynamic
lib? static?
How can I link it against the dynamic? *.so?

*.a is the static library (an archive with lots of *.o files inside).
*.so is the shared object file for dynamic linking.
"mpic++ -o mpi-test mpi-test.cpp
/opt/library/boost/1.39.0/lib/libboost_mpi*.so"

perfekt. Now the mpirun searches the *.so

Post by Michael Hofmann
"export LIBRARY_PATH=$BOOST/lib:$LIBRARY_PATH"

No, in our systemconfig /etc/profile we use also LD_LIBRARY_PATH

LD_LIBRARY_PATH is used by the "loader" to search for *.so files when
then program is executed.
LIBRARY_PATH is used by the compiler/linker to search for library files
(both *.a and *.so). If LIBRARY_PATH is set correctly, you can use the
(short) option "-lboost_mpi-gcc42-mt" to link against your Boost MPI
library (dynamic linking unless otherwise specified, e.g. with
"-static"). In this case, the linker finds the library file by using
LIBRARY_PATH and the name used in the "-l<name>" option.
As an alternative, you can also specify the full path to the library
file manually (/opt/library/boost/1.39.0/lib/libboost_mpi...), e.g. to
test whether linking to this file is possible or not.

okay, I have set the OMPI_LDFLAGS with the -L and -l switches so i
don't need de LIBRARY_PATH

Thanks, now I have the problem, 2 systems, same config and same code.
If I run the compiled code with mpirun I get:
mpi-test: error while loading shared libraries:
libboost_mpi-gcc42-mt-1_39.so.1.39.0: cannot open shared object file:
No such file or directory

LD Path is set to LD_LIBRARY_PATH=/<myPath>/library/boost/1.39.0/lib,
file is there. Static linking works (*.a file)

Thanks Phil

Michael Hofmann

2009-06-10 14:01:17 UTC

Am 10.06.2009, 14:47 Uhr, schrieb Philipp Kraus

okay, I have set the OMPI_LDFLAGS with the -L and -l switches so i don't
need de LIBRARY_PATH
Thanks, now I have the problem, 2 systems, same config and same code. If
libboost_mpi-gcc42-mt-1_39.so.1.39.0: cannot open shared object file: No
such file or directory
LD Path is set to LD_LIBRARY_PATH=/<myPath>/library/boost/1.39.0/lib,
file is there. Static linking works (*.a file)

Is LD_LIBRARY_PATH set on both systems?

Is the libboost_mpi*.so* file available on both systems, with the same
name?

Check the output of "ldd mpi-test", is the libboost_mpi*.so file found on
both systems?

What MPI implementation are you using?

"/etc/profile" is used only for login shells, maybe LD_LIBRARY_PATH is not
set correctly for the (remote) MPI processes. With some mpirun commands
(e.g. OpenMPI) you can also specify environment variables manually.

Instead of using LD_LIBRARY_PATH, you can also add the
"/opt/library/boost/1.39.0/lib" to the configuration files of ld.so (in
/etc/ld.so.conf or by creating a separate conf-file in
/etc/ld.so.conf.d/). After that, you need to update the shared object
cache with "ldconfig". With "ldconfig -v" you can verify that the
libboost_mpi*.so file is found. If this works, you do not need to set
LD_LIBRARY_PATH (on that system) anymore.

Michael

Philipp Kraus

2009-06-10 15:41:09 UTC

On 2009-06-10 16:01:17 +0200, "Michael Hofmann"

Post by Philipp Kraus
okay, I have set the OMPI_LDFLAGS with the -L and -l switches so i
don't need de LIBRARY_PATH
Thanks, now I have the problem, 2 systems, same config and same code.
No such file or directory
LD Path is set to LD_LIBRARY_PATH=/<myPath>/library/boost/1.39.0/lib,
file is there. Static linking works (*.a file)

Is LD_LIBRARY_PATH set on both systems?
Is the libboost_mpi*.so* file available on both systems, with the same name?

yes, because they are all in my homedir and /home is mounted via NFS on
all cluster machines and
the machines are cloned.

Post by Michael Hofmann
Check the output of "ldd mpi-test", is the libboost_mpi*.so file found
on both systems?

ldd shows:
linux-vdso.so.1 => (0x00007fff563fe000)
libboost_mpi-gcc42-mt-1_39.so.1.39.0 =>
/home/<user>/library/boost/1.39.0/lib/libboost_mpi-gcc42-mt-1_39.so.1.39.0
(0x00007faf4ddb4000)
libmpi_cxx.so.0 => /usr/lib/libmpi_cxx.so.0 (0x00007faf4db91000)
libmpi.so.0 => /usr/lib/libmpi.so.0 (0x00007faf4d8f8000)
libopen-rte.so.0 => /usr/lib/libopen-rte.so.0 (0x00007faf4d69b000)
libopen-pal.so.0 => /usr/lib/libopen-pal.so.0 (0x00007faf4d43a000)
libdl.so.2 => /lib/libdl.so.2 (0x00007faf4d236000)
libnsl.so.1 => /lib/libnsl.so.1 (0x00007faf4d01d000)
libutil.so.1 => /lib/libutil.so.1 (0x00007faf4ce1a000)
libstdc++.so.6 => /usr/lib/libstdc++.so.6 (0x00007faf4cb0f000)
libm.so.6 => /lib/libm.so.6 (0x00007faf4c88e000)
libgcc_s.so.1 => /lib/libgcc_s.so.1 (0x00007faf4c680000)
libpthread.so.0 => /lib/libpthread.so.0 (0x00007faf4c464000)
libc.so.6 => /lib/libc.so.6 (0x00007faf4c102000)
libboost_serialization-gcc42-mt-1_39.so.1.39.0 =>
/home/pkr/library/boost/1.39.0/lib/libboost_serialization-gcc42-mt-1_39.so.1.39.0
(0x00007faf4be8e000)
librt.so.1 => /lib/librt.so.1 (0x00007faf4bc85000)
/lib64/ld-linux-x86-64.so.2 (0x00007faf4dff3000)

Post by Michael Hofmann
What MPI implementation are you using?

I don't know exactly, because I don't have admin rights. It's the last
LTS Ubuntu Version

Post by Michael Hofmann
"/etc/profile" is used only for login shells, maybe LD_LIBRARY_PATH is
not set correctly for the (remote) MPI processes. With some mpirun
commands (e.g. OpenMPI) you can also specify environment variables
manually.

I can't modify /etc/profile (no rights), I have modified my ~/.profile
whitch is shared via NFS to the other machines. If I connect with SSH,
I can see the same config (and errors)

Post by Michael Hofmann
Instead of using LD_LIBRARY_PATH, you can also add the
"/opt/library/boost/1.39.0/lib" to the configuration files of ld.so (in
/etc/ld.so.conf or by creating a separate conf-file in
/etc/ld.so.conf.d/). After that, you need to update the shared object
cache with "ldconfig". With "ldconfig -v" you can verify that the
libboost_mpi*.so file is found. If this works, you do not need to set
LD_LIBRARY_PATH (on that system) anymore.

With ldconfig -v I can't find my boost libs. On my Gentoo it runs now
always fine, but not on the Ubuntu.
I must configure Boostlib only local for my user

Thanks again

Phil

Michael Hofmann

2009-06-11 07:34:01 UTC

Am 10.06.2009, 17:41 Uhr, schrieb Philipp Kraus

Post by Philipp Kraus
yes, because they are all in my homedir and /home is mounted via NFS on
all cluster machines and
the machines are cloned.

OK, I thought you have root access on all machines.

Post by Michael Hofmann
What MPI implementation are you using?

I don't know exactly, because I don't have admin rights. It's the last
LTS Ubuntu Version

"mpirun -V"

Post by Philipp Kraus
I can't modify /etc/profile (no rights), I have modified my ~/.profile
whitch is shared via NFS to the other machines. If I connect with SSH, I
can see the same config (and errors)

With "config" you mean setting LD_LIBRARY_PATH in "~/.profile"? What
"errors" occur if you connect with SSH?

First, you have to make sure that all is correct if you connect with SSH!
The test with "ldd mpi-test" should succeed on all machines you want to
use!

Then you should investigate why it is not working when starting remote
processes with mpirun:
- check the output of "mpirun -np 2 ldd mpi-test"
- verify that the LD_LIBRARY_PATH is set correctly, e.g. by starting the
following shell script with mpirun
------------------------------------
#!/bin/sh
export > export.`hostname`.${RANDOM}
------------------------------------

Does it work if you copy the required Boost shared objects in the same
directory as the MPI test program?

Michael

Philipp Kraus

2009-06-11 10:12:51 UTC

On 2009-06-11 09:34:01 +0200, "Michael Hofmann"

Post by Philipp Kraus
yes, because they are all in my homedir and /home is mounted via NFS on
all cluster machines and
the machines are cloned.

OK, I thought you have root access on all machines.

not yet, If I finished the installation, I can talk to my prof for the
installation in the system

Post by Michael Hofmann
What MPI implementation are you using?

I don't know exactly, because I don't have admin rights. It's the last
LTS Ubuntu Version

"mpirun -V"

mpirun (Open MPI) 1.2.5

With "config" you mean setting LD_LIBRARY_PATH in "~/.profile"?

Here my library Path (shown with env on all machines)

LIBRARY_PATH=/home/pkr/library/gsl/1.9.0/lib:/home/pkr/library/boost/1.39.0/lib:/home/pkr/library/gsl/1.9.0/lib:/home/pkr/library/boost/1.39.0/lib:

LD_LIBRARY_PATH=/home/pkr/library/gsl/1.9.0/lib:/home/pkr/library/boost/1.39.0/lib:/home/pkr/library/gsl/1.9.0/lib:/home/pkr/library/boost/1.39.0/lib:/opt/intel/Compiler/11.0/081/lib/intel64/:

OMPI_LDFLAGS=-lboost_mpi-gcc42-mt-1_39

-lboost_serialization-gcc42-mt-1_39 -lboost_mpi-gcc42-mt-1_39
-lboost_serialization-gcc42-mt-1_39

CPATH=/home/pkr/library/gsl/1.9.0/include:/home/pkr/library/boost/1.39.0/include/boost-1_39:/home/pkr/library/gsl/1.9.0/include:/home/pkr/library/boost/1.39.0/include/boost-1_39:

What

Post by Michael Hofmann
"errors" occur if you connect with SSH?

My SSH Login via keyfiles on all machines don't shows an error on login
(I can't read the /var/log to get information about the ssh logfiles)

Post by Michael Hofmann
First, you have to make sure that all is correct if you connect with
SSH! The test with "ldd mpi-test" should succeed on all machines you
want to use!

ldd show on all machines:

linux-vdso.so.1 => (0x00007fff7dffe000)
libboost_mpi-gcc42-mt-1_39.so.1.39.0 =>
/home/pkr/library/boost/1.39.0/lib/libboost_mpi-gcc42-mt-1_39.so.1.39.0
(0x00007fc675aed000)
libboost_serialization-gcc42-mt-1_39.so.1.39.0 =>
/home/pkr/library/boost/1.39.0/lib/libboost_serialization-gcc42-mt-1_39.so.1.39.0
(0x00007fc675879000)
libmpi_cxx.so.0 => /usr/lib/libmpi_cxx.so.0 (0x00007fc675656000)
libmpi.so.0 => /usr/lib/libmpi.so.0 (0x00007fc6753bd000)
libopen-rte.so.0 => /usr/lib/libopen-rte.so.0 (0x00007fc675160000)
libopen-pal.so.0 => /usr/lib/libopen-pal.so.0 (0x00007fc674eff000)
libdl.so.2 => /lib/libdl.so.2 (0x00007fc674cfb000)
libnsl.so.1 => /lib/libnsl.so.1 (0x00007fc674ae2000)
libutil.so.1 => /lib/libutil.so.1 (0x00007fc6748df000)
libstdc++.so.6 => /usr/lib/libstdc++.so.6 (0x00007fc6745d4000)
libm.so.6 => /lib/libm.so.6 (0x00007fc674353000)
libgcc_s.so.1 => /lib/libgcc_s.so.1 (0x00007fc674145000)
libpthread.so.0 => /lib/libpthread.so.0 (0x00007fc673f29000)
libc.so.6 => /lib/libc.so.6 (0x00007fc673bc7000)
librt.so.1 => /lib/librt.so.1 (0x00007fc6739be000)
/lib64/ld-linux-x86-64.so.2 (0x00007fc675d2c000)

Post by Michael Hofmann
Then you should investigate why it is not working when starting remote
- check the output of "mpirun -np 2 ldd mpi-test"

I get lot of these errors:
mpi-test: error while loading shared libraries:
libboost_mpi-gcc42-mt-1_39.so.1.39.0: cannot open shared object file:
No such file or directory

Post by Michael Hofmann
- verify that the LD_LIBRARY_PATH is set correctly, e.g. by starting
the following shell script with mpirun
------------------------------------
#!/bin/sh
export > export.`hostname`.${RANDOM}
------------------------------------

That works.

Post by Michael Hofmann
Does it work if you copy the required Boost shared objects in the same
directory as the MPI test program?

No the same error "loading shared lib.."

Is there an writing error in my config or anything else? My process
can't find the Boost libs, okay, but why? LD for linking and
LIBRARY pathes are set correctly

Thanks

Phil

Michael Hofmann

2009-06-11 13:54:52 UTC

Am 11.06.2009, 12:12 Uhr, schrieb Philipp Kraus

Post by Michael Hofmann
"mpirun -V"

mpirun (Open MPI) 1.2.5

OK, to make a long story short: it seems that OpenMPI totally ignores the
variables that are set in /etc/profile or in ~/.profile.

Therefore, the solution for you is to tell mpirun to export the
LD_LIBRARY_PATH variable to the remote processes. This can be achieved
with "-x" option:

mpirun -x LD_LIBRARY_PATH=/home/pkr/library/boost/1.39.0/lib -np 2
mpi-test

If LD_LIBRARY_PATH is already set on the local machine, it suffices to use:

mpirun -x LD_LIBRARY_PATH -np 2 mpi-test

Michael

Philipp Kraus

2009-06-11 18:35:41 UTC

On 2009-06-11 15:54:52 +0200, "Michael Hofmann"