help with allgatherv

[m***@hotmail.com]

Hello All,

I have been struggling with numerous access violations when I use
allgatherv to reassamble a matrix row by row as follows (in
fortran90). I am trying to gather local row ilx and put it into global
row igx, for example: I have a local matrix on each proc that is
100x1000. The global matrix is 200x1000. The local matrix on proc 0
is the first halft (rows 1:100) of the global matrix and the on proc 1
it is the 2nd half of the global matrix (rows 101:200)

! Local matrix
allocate( lmat( 100, 1000))
allocate( gmat( 200, 1000))

.......

call MPI_TYPE_VECTOR( 100, 1, 1000, MPI_DOUBLE_COMPLEX, local_type,
ierr)
call MPI_TYPE_COMMIT( local_type,ierr)

call MPI_TYPE_VECTOR( 200, 1, 1000, MPI_DOUBLE_COMPLEX, global_type,
ierr)
call MPI_TYPE_COMMIT( global_type,ierr)


! The following test of gathering the 44th local row into the global
matrix, that is proc 0 row goes to global row 44, and proc 1 row goes
to global row 144.

counts = (/1, 1/)
disps = (/0, 100/)

call MPI_ALLGATHERv( lmat(44,1), 1, local_type, gmat(1,1), counts,
disps, globa_type, MPI_COMM_WORLD,ierr)

I run the code and get:

% forrtl: severe (157): Program Exception - access violation

Using send and recv works:

if (iam == 1) then
call MPI_SEND( lmat(44,1), 1, local_type, 0 , 1, MPI_COMM_WORLD,
ierr)
else
call MPI_RECV( gmat(144,1), 1, global_type, 1 , 1, MPI_COMM_WORLD,
status_array, ierr)
end if

Thanks all

[Michael Hofmann]

Post by m***@hotmail.com
Hello All,
I have been struggling with numerous access violations when I use
allgatherv to reassamble a matrix row by row as follows (in
fortran90). I am trying to gather local row ilx and put it into global
row igx, for example: I have a local matrix on each proc that is
100x1000. The global matrix is 200x1000. The local matrix on proc 0
is the first halft (rows 1:100) of the global matrix and the on proc 1
it is the 2nd half of the global matrix (rows 101:200)
! Local matrix
allocate( lmat( 100, 1000))
allocate( gmat( 200, 1000))
.......
call MPI_TYPE_VECTOR( 100, 1, 1000, MPI_DOUBLE_COMPLEX, local_type,

^^^^^^^^^^^^
One row of the local 100x1000 matrix has 1000 elements!?! Therefore, the
construction of the local type should look like this: MPI_TYPE_VECTOR(
1000, 1, 100, ...)

Post by m***@hotmail.com
ierr)
call MPI_TYPE_COMMIT( local_type,ierr)
call MPI_TYPE_VECTOR( 200, 1, 1000, MPI_DOUBLE_COMPLEX, global_type,
ierr)
call MPI_TYPE_COMMIT( global_type,ierr)

The receive type of MPI_Allgatherv has to match the type of the data from
_each_ process, not the type of the global result. From this point of
view, the receive type is the same as the send type (local_type).

Post by m***@hotmail.com
! The following test of gathering the 44th local row into the global
matrix, that is proc 0 row goes to global row 44, and proc 1 row goes
to global row 144.
counts = (/1, 1/)
disps = (/0, 100/)
call MPI_ALLGATHERv( lmat(44,1), 1, local_type, gmat(1,1), counts,

^^^^^^^^^^ ^^^^^^^^^
gmat(44,1)?

Post by m***@hotmail.com
disps, globa_type, MPI_COMM_WORLD,ierr)

^^^^^ ^^^^^^^^^^

Because of the column-major ordering in Fortran, the correct displacements
are 0 * real_size_of_MPI_DOUBLE_COMPLEX and 100 *
real_size_of_MPI_DOUBLE_COMPLEX. However, displacements are given in units
of the extent of the receive type. Neither local_type nor global_type have
the correct extent (real_size_of_MPI_DOUBLE_COMPLEX), because the default
extent of a datatype in MPI is calculated from "the span from the first
byte to the last byte occupied by entries in this datatype".

To make it short: You need a new receive type that represents a row (1000
elements with stride 100!), but has an extent of only 1 element. MPI-2.1
has the new function "MPI_TYPE_CREATE_RESIZED" for that. Example 4 on this
page http://www.mpi-forum.org/docs/mpi21-report-bw/node81.htm#Node81 shows
the same for older versions of MPI.


Michael

[James]

Thanks for your reply Michael,

I'm still having trouble with what I think should be straight forward
- thanks for your patients.
I read the example link you provided and tried the following example
(posted below) to no avail.
I can't seem to figure out how to gather from row to row, instead the
code goes row to column.
Also, for some reason only processor 0 gathers, here is the output:

0 : 1.000000 2.000000 3.000000
0 : 2.100000 4.100000 6.100000
-----------
1 : 10.00000 20.00000 30.00000
1 : 21.00000 41.00000 61.00000
-----------
1 row 1 1.000000 2.000000 3.000000
1 row 2 2.100000 4.100000 6.100000
1 row 3 0.0000000E+00 0.0000000E+00 0.0000000E
+00
1 row 4 0.0000000E+00 0.0000000E+00 0.0000000E
+00
0 row 1 1.000000 2.000000 3.000000
0 row 2 2.100000 4.100000 6.100000
0 row 3 0.0000000E+00 0.0000000E+00 0.0000000E
+00
0 row 4 0.0000000E+00 0.0000000E+00 0.0000000E
+00

======================================================================

program mpi_test

implicit none

include 'mpif.h'

integer :: iam ! processor rank [0,np-1]
integer :: np ! number of processors
integer :: ierr ! exit error code from MPI routines
integer :: ipln ! length of processor name string
character(len=150) :: ipName ! processor name

double precision, dimension(2,3) :: a
double precision, dimension(3,4) :: c
double precision, dimension(4,3) :: b

integer :: sizeofdouble, i, row, row1
integer, dimension(2) :: disp, otype, blocklen

call MPI_INIT(ierr)
call MPI_COMM_RANK(MPI_COMM_WORLD,iam,ierr)
call MPI_COMM_SIZE(MPI_COMM_WORLD,np,ierr)
call MPI_GET_PROCESSOR_NAME(ipName,ipln,ierr)

if (iam == 0) then

a(1,:) = (/1.0d0, 2.0d0, 3.0d0/)
a(2,:) = (/2.1d0, 4.1d0, 6.1d0/)

else

a(1,:) = (/10.0d0, 20.0d0, 30.0d0/)
a(2,:) = (/21.0d0, 41.0d0, 61.0d0/)

end if

b = 0.0d0
do i=1,2
write(*,*) iam,':',sngl(a(i,:))
end do
write(*,*) '-----------'

CALL MPI_TYPE_EXTENT( MPI_DOUBLE_PRECISION, sizeofdouble, ierr)

CALL MPI_TYPE_VECTOR( 3, 1, 2, MPI_DOUBLE_PRECISION, row, ierr)

! create datatype for one row, with the extent of one double number
disp(1) = 0
disp(2) = sizeofdouble
otype(1) = row
otype(2) = MPI_UB
blocklen(1) = 1
blocklen(2) = 1
CALL MPI_TYPE_STRUCT( 2, blocklen, disp, otype, row1, ierr)

CALL MPI_TYPE_COMMIT( row1, ierr)

CALL MPI_ALLGATHER(a,2,row1,c,
4*3,MPI_DOUBLE_PRECISION,MPI_COMM_WORLD,ierr)

b = transpose(c)
do i=1,4
write(*,*) iam,'row',i,sngl(b(i,:))
end do

call MPI_BARRIER(MPI_COMM_WORLD,ierr)
call MPI_FINALIZE(ierr)

end program mpi_test

Post by m***@hotmail.com
Hello All,
I have been struggling with numerous access violations when I use
allgatherv to reassamble a matrix row by row as follows (in
fortran90). I am trying to gather local row ilx and put it into global
row igx, for example: I have a local matrix on each proc that is
100x1000.  The global matrix is 200x1000.  The local matrix on proc 0
is the first halft (rows 1:100) of the global matrix and the on proc 1
it is the 2nd half of the global matrix (rows 101:200)
! Local matrix
allocate( lmat( 100, 1000))
allocate( gmat( 200, 1000))
.......
call MPI_TYPE_VECTOR( 100, 1, 1000, MPI_DOUBLE_COMPLEX, local_type,

                         ^^^^^^^^^^^^
One row of the local 100x1000 matrix has 1000 elements!?! Therefore, the  
construction of the local type should look like this: MPI_TYPE_VECTOR(  
1000, 1, 100, ...)

Post by m***@hotmail.com
ierr)
call MPI_TYPE_COMMIT( local_type,ierr)
call MPI_TYPE_VECTOR( 200, 1, 1000, MPI_DOUBLE_COMPLEX, global_type,
ierr)
call MPI_TYPE_COMMIT( global_type,ierr)

The receive type of MPI_Allgatherv has to match the type of the data from  
_each_ process, not the type of the global result. From this point of  
view, the receive type is the same as the send type (local_type).

Post by m***@hotmail.com
! The following test of gathering the 44th local row into the global
matrix, that is proc 0 row goes to global row 44, and proc 1 row goes
to global row 144.
counts = (/1, 1/)
disps = (/0, 100/)
call MPI_ALLGATHERv( lmat(44,1), 1, local_type, gmat(1,1), counts,

                        ^^^^^^^^^^                 ^^^^^^^^^
gmat(44,1)?

Post by m***@hotmail.com
disps, globa_type, MPI_COMM_WORLD,ierr)

   ^^^^^  ^^^^^^^^^^
Because of the column-major ordering in Fortran, the correct displacements  
are 0 * real_size_of_MPI_DOUBLE_COMPLEX and 100 *  
real_size_of_MPI_DOUBLE_COMPLEX. However, displacements are given in units  
of the extent of the receive type. Neither local_type nor global_type have  
the correct extent (real_size_of_MPI_DOUBLE_COMPLEX), because the default  
extent of a datatype in MPI is calculated from "the span from the first  
byte to the last byte occupied by entries in this datatype".
To make it short: You need a new receive type that represents a row (1000  
elements with stride 100!), but has an extent of only 1 element. MPI-2.1  
has the new function "MPI_TYPE_CREATE_RESIZED" for that. Example 4 on this  
pagehttp://www.mpi-forum.org/docs/mpi21-report-bw/node81.htm#Node81shows  
the same for older versions of MPI.
Michael- Hide quoted text -
- Show quoted text -

[Michael Hofmann]

Post by James
I'm still having trouble with what I think should be straight forward
- thanks for your patients.
I read the example link you provided and tried the following example
(posted below) to no avail.
...
CALL MPI_ALLGATHER(a,2,row1,c,
4*3,MPI_DOUBLE_PRECISION,MPI_COMM_WORLD,ierr)

^^^
Change "4*3" to "2*3" and it works. The receive count of MPI_Allgather
specifies the "number of elements received from any process", _not_ the
total number of elements to receive!

Post by James
b = transpose(c)

You are receiving the data in row-major order and transform it to the
desired column-major order. This is an additional step and requires an
additional buffer. The example below shows how to omit this extra step.

program mpi_test
...
...
integer :: row_global, row_global1
...
...

! create a second row datatype with stride = 4 (stride of receive buffer
"b")
CALL MPI_TYPE_VECTOR( 3, 1, 4, MPI_DOUBLE_PRECISION, row_global, ierr)
otype(1) = row_global
CALL MPI_TYPE_STRUCT( 2, blocklen, disp, otype, row_global1, ierr)
CALL MPI_TYPE_COMMIT( row_global1, ierr)

CALL MPI_ALLGATHER(a,2,row1,b,2,row_global1,MPI_COMM_WORLD,ierr)

do i=1,4
write(*,*) iam,'row',i,sngl(b(i,:))
end do

...
...
end program mpi_test


Michael