Programming of Parallel Computers/Lecture 4
From PBDN
This content is a student-produced work-in-progress and may be incomplete or contain errors.
You can help by adding new material, links, or by correcting errors.
A wiki is a collaborative resource. The idea here is that a valuable communal resource can be created if many of us contribute to it, not that one student should create a resource for your benefit. If you don't think that you can make a direct positive contribution right now, that's OK, but at least register an account to signify that you are willing to help.
Contents |
MPI
Point-to-point communication
Continued from Lecture 3
Standard send
With a standard send, the implementation chooses between synchronous send and buffered send.
For "small" messages, where "small" means having size less than some implementation-defined limit, MPI uses a buffered send using an internal buffer.
For "large" messages (size greater than the limit), a synchronous send is used.
Programming advice
- Post a non-blocking receive as soon as possible
- Overlap communications with computation as much as possible
Other MPI calls
- MPI_Sendrecv - send and receive in one call
- MPI_Sendrecv_replace - as above, but replace already-sent data with received data.
Derived datatypes
Suppose that we want to send one column of a nx * ny matrix with data stored row-wise in a simple array:
double *A = new double[nx*ny];
Specifically, say we want to send the last column of a matrix from P0 to P1. For P1, it is the first column of a matrix of the same size. This situation occurs commonly where a matrix is distributed over many processors.
The elements of a column are not stored contiguously in memory, so how do we send the column from P0 to P1?
Send elements one-at-a-time
This is a very poor solution. The reason is that there is a latency, or setup overhead, associated with each communication. The total communication time, Tc, to send a message m bytes long is Tc = Ts + mTb, where Ts is the setup time (latency) and Tb is the time to send one byte (the reciprocal of the bandwidth available between the processors).
To send a given amount of data, we want to send a small number of big messages, not many small messages.
Copy elements to a contiguous array and send that
Use MPI's derived datatypes
- MPI_Type_vector
- MPI_Type_commit
- MPI_Type_free
Other MPI derived datatypes
In addition to MPI_Type_vector, there are
- MPI_Type_indexed
- MPI_Type_struct
Collective operations
Continued in Lecture 5.