1
What is MPI?

MPI = Message Passing Interface

Specification of message passing libraries for
developers and users

Not a library by itself, but specifies what such a library
should be

Specifies application programming interface (API) for
such libraries

Many libraries implement such APIs on different
platforms – MPI libraries

Goal: provide a standard for writing message
passing programs

Portable, efficient, flexible

Language binding: C, C++, FORTRAN programs
2
History & Evolution

1980s – 1990s: incompatible
libraries and software tools; need
for a standard

1994, MPI 1.0;


1995, MPI 1.1, revision and
clarification to MPI 1.0

Major milestone

C, FORTRAN

Fully implemented in all MPI
libraries

1997, MPI 1.2

Corrections and clarifications to
MPI 1.1

1997, MPI 2

Major extension (and clarifications)
to MPI 1.1

C++, C, FORTRAN

Partially implemented in most
libraries; a few full implementations
(e.g. ANL MPICH2)
MPI Evolution
3
Why Use MPI?


Standardization: de facto standard for parallel
computing

Not an IEEE or ISO standard, but “industry standard”

Practically replaced all previous message passing
libraries

Portability: supported on virtually all HPC
platforms

No need to modify source code when migrating to
different machines

Performance: so far the best; high performance
and high scalability

Rich functionality:

MPI 1.1 – 125 functions

MPI 2 – 152 functions.
If you know 6 MPI functions,
you can do almost everything
in parallel.
4
Programming Model

Message passing model: data exchange through explicit
communications.


For distributed memory, as well as shared-memory
parallel machines

User has full control (data partition, distribution): needs
to identify parallelism and implement parallel algorithms
using MPI function calls.

Number of CPUs in computation is static

New tasks cannot be dynamically spawned during run time (MPI
1.1)

MPI 2 specifies dynamic process creation and management, but
not available in most implementations.

Not necessarily a disadvantage

General assumption: one-to-one mapping of MPI
processes to processors (although not necessarily
always true).
5
MPI 1.1 Overview

Point to point communications

Collective communications

Process groups and communicators


Process topologies

MPI environment management
6
MPI 2 Overview

Dynamic process creation and management

One-sided communications

MPI Input/Output (Parallel I/O)

Extended collective communications

C++ binding
7
MPI Resources

MPI Standard:

/>
MPI web sites/tutorials etc, see class web site

Public domain (free) MPI implementations

MPICH and MPICH2 (from ANL)

LAM MPI
8
General MPI Program Structure

9
Example
#include <mpi.h>
#include <stdio.h>
int main(int argc, char **argv)
{
int my_rank, num_cpus;
MPI_Init(&argc, &argv);
MPI_Comm_rank(MPI_COMM_WORLD, &my_rank);
MPI_Comm_size(MPI_COMM_WORLD, &num_cpus);
printf(“Hello, I am process %d among %d processes\n”,
my_rank, num_cpus);
MPI_Finalize();
return 0;
}
Hello, I am process 1 among 4 processes
Hello, I am process 2 among 4 processes
Hello, I am process 0 among 4 processes
Hello, I am process 3 among 4 processes
On 4 processors:
10
Example
program hello
implicit none
include ‘mpif.h’
integer :: ierr, my_rank, num_cpus
call MPI_INIT(ierr)
call MPI_COMM_RANK(MPI_COMM_WORLD, my_rank)
call MPI_COMM_SIZE(MPI_COMM_WORLD, num_cpus)
write(*,*) “Hello, I am process “, my_rank, “ among “ &

, num_cpus, “ processes”
call MPI_FINALIZE(ierr)
end program hello
Hello, I am process 1 among 4 processes
Hello, I am process 2 among 4 processes
Hello, I am process 0 among 4 processes
Hello, I am process 3 among 4 processes
On 4 processors:
11
MPI Header Files

In C/C++:

In FORTRAN:
#include <mpi.h>
include ‘mpif.h’
or (in FORTRAN90 and later)
use MPI
12
MPI Naming Conventions

All names have MPI_ prefix.

In FORTRAN:

All subroutine names upper case, last argument is return code

A few functions without return code

In C: mixed uppercase/lowercase


MPI constants all uppercase
call MPI_XXXX(arg1,arg2,…,ierr)
call MPI_XXXX_XXXX(arg1,arg2,…,ierr)
ierr = MPI_Xxxx(arg1,arg2,…);
ierr = MPI_Xxxx_xxx(arg1,arg2,…);
MPI_COMM_WORLD, MPI_SUCCESS, MPI_DOUBLE, MPI_SUM, …
If ierr == MPI_SUCCESS,
Everything is ok; otherwise,
something is wrong.
13
Initialization

Initialization: MPI_Init() initializes MPI environment;
(MPI_Init_thread() if multiple threads)

Must be called before any other MPI routine (so put it at the
beginning of code) except MPI_Initialized() routine.

Can be called only once; subsequent calls are erroneous.

MPI_Initialized() to check if MPI_Init() is called
int main(int argc, char ** argv)
{
MPI_Init(&argc, &argv);
int flag;
MPI_Initialized(&flag);
if(flag != 0) … // MPI_Init called
… …
MPI_Finalize();

return 0;
}
int MPI_Init(int *argc, char ***argv)
program test
integer ierr
call MPI_INIT(ierr)
…
call MPI_FINALIZE(ierr)
end program test
MPI_INIT(ierr)
14
Termination

MPI_Finalize() cleans up MPI environment

Must be called before exits.

No other MPI routine can be called after this call,
even MPI_INIT()

Exception: MPI_Initialized() (and
MPI_Get_version(), MPI_Finalized()).

Abnormal termination: MPI_Abort()

Makes a best attempt to abort all tasks
int MPI_Finalize(void)
MPI_FINALIZE(IERR)
integer IERR
int MPI_Abort(MPI_Comm comm, int errorcode)

MPI_ABORT(COMM,ERRORCODE,IERR)
integer COMM, ERRORCODE, IERR
15
MPI Processes

MPI is process-oriented: program consists of multiple
processes, each corresponding to one processor.

MIMD: Each process runs its own code. In practice, runs
its own copy of the same code (SPMD).

MPI process and threads: MPI process can contain a
single thread (common case) or multiple threads.

Most MPI implementations do not support multiple threads.
Needs special processing with that support.

We will assume a single thread per process from now on.

MPI processes are identified by their ranks:

If total nprocs processes in computation, rank ranges from 0,
1, …, nprocs-1. (true in C and FORTRAN).

nprocs does not change during computation.
16
Communicators and
Process Groups

Communicator: is a group of processes that can

communicate with one another.

Most MPI routines require a communicator argument to
specify the collection of processes the communication is
based on.

All processes in the computation form the communicator
MPI_COMM_WORLD.

MPI_COMM_WORLD is pre-defined by MPI, available anywhere

Can create subgroups/subcommunicators within
MPI_COMM_WORLD.

A process may belong to different communicators, and have
different ranks in different communicators.
17
How many CPUs, Which one am I …

How many CPUs: MPI_COMM_SIZE()

Who am I: MPI_COMM_RANK()

Can compute data decomposition etc.

Know total number of grid points, total number of cpus and
current cpu id; can calc which portion of data current cpu is to
work on.

E.g. Poisson equ on a square


Ranks also used to specify source and destination of
communications.
…
int my_rank, ncpus;
MPI_Comm_rank(MPI_COMM_WORLD, &my_rank);
MPI_Comm_size(MPI_COMM_WORLD, &ncpus);
…
int MPI_Comm_rank(MPI_Comm comm, int *rank)
int MPI_Comm_size(MPI_Comm comm, int *size)
MPI_COMM_RANK(comm,rank,ierr)
MPI_COMM_SIZE(comm,size,ierr)
my_rank value different on different processors !
18
Compiling, Running

MPI standard does not specify how to start up the
program

Compiling and running MPI code implementation dependent

MPI implementations provide utilities/commands for
compiling/running MPI codes

Compile: mpicc, mpiCC, mpif77, mpif90, mpCC, mpxlf …
mpiCC –o myprog myfile.C (cluster)
mpif90 –o myprog myfile.f90 (cluster)
CC –Ipath_mpi_include –o myprog myfile.C –lmpi (SGI)
mpCC –o myprog myfile.C (IBM)


Run: mpirun, poe, prun, ibrun …
mpirun –np 2 myprog (cluster)
mpiexec –np 2 myprog (cluster)
poe myprog –node 1 –tasks_per_node 2 … (IBM)
19
Example
#include <mpi.h>
#include <stdio.h>
#include <string.h>
int main(int argc, char **argv)
{
char message[256];
int my_rank;
MPI_Status status;
MPI_Init(&argc, &argv);
MPI_Comm_rank(MPI_COMM_WORLD,&my_rank);
if(my_rank==0){
strcpy(message,”Hello, there!”);
MPI_Send(message,strlen(message)+1,MPI_CHAR,1,99,MPI_COMM_WORLD);
}
else if(my_rank==1) {
MPI_Recv(message,256,MPI_CHAR,0,99,MPI_COMM_WORLD,&status);
printf(“Process %d received: %s\n”,my_rank,message);
}
MPI_Finalize();
return 0;
}
mpirun –np 2 test_hello
Process 1 received: Hello, there!
6 MPI functions:

MPI_Init()
MPI_Finalize()
MPI_Comm_rank()
MPI_Comm_size()
MPI_Send()
MPI_Recv()
20
MPI Communications

Point-to-point communications

Involves a sender and a receiver, one processor to another
processor

Only the two processors participate in communication

Collective communications

All processors within a communicator participate in
communication (by calling same routine, may pass different
arguments);

Barrier, reduction operations, gather, …
21
Point-to-Point Communications
22
Send / Receive

Message data: what to send/receive?


Where is the message? Where to put it?

What kind of data is it? What is the size?

Message envelope: where to send/receive?

Sender, receiver

Communication context

Message tag.
…
MPI_Send(message,strlen(message)+1,MPI_CHAR,1,99,MPI_COMM_WORLD);
MPI_Recv(message,256,MPI_CHAR,0,99,MPI_COMM_WORLD,&status);
…
23
Send

buf – memory address of start of message

count – number of data items

datatype – what type each data item is (integer,
character, double, float …)

dest – rank of receiving process

tag – additional identification of message

comm – communicator, usually MPI_COMM_WORLD

int MPI_Send(void *buf,int count,MPI_Datatype datatype,
int dest, int tag, MPI_Comm comm)
MPI_SEND(BUF,COUNT,DATATYPE,DEST,TAG,COMM,IERROR)
<type>BUF(*)
integer COUNT,DATATYPE,DEST,TAG,COMM,IERROR
char message[256];
MPI_Send(message,strlen(message)+1,MPI_CHAR,1,99,MPI_COMM_WORLD);
24
Receive

buf – initial address of receive buffer

count – number of elements in receive buffer (size of receive buffer)

may not equal to the count of items actually received

Actual number of data items received can be obtained by calling
MPI_Get_count().

datatype – data type in receive buffer

source – rank of sending process

tag – additional identification for message

comm – communicator, usually MPI_COMM_WORLD

status – object containing additional info of received message

ierror – return code

int MPI_Recv(void *buf,int count,MPI_Datatype datatype,int source,int tag,
MPI_Comm comm,MPI_Status *status)
MPI_RECV(BUF,COUNT,DATATYPE,SOURCE,TAG,COMM,STATUS,IERROR)
<type>BUF(*)
integer COUNT,DATATYPE,SOURCE,TAG,COMM,STATUS(MPI_STATUS_SIZE),IERROR
Actual number of data items received can be queried from status object; it may be
smaller than count, but cannot be larger (if larger  overflow error).
char message[256];
MPI_Recv(message,256,MPI_CHAR,0,99,MPI_COMM_WORLD,&status);
25
MPI_Recv Status

In C: MPI_Status structure, 3 members; MPI_Status status

status.MPI_TAG – tag of received message

status.MPI_SOURCE – source rank of message

status.MPI_ERROR – error code

In FORTRAN: integer array; integer status(MPI_STATUS_SIZE)

Status(MPI_TAG) – tag of received message

status(MPI_SOURCE) – source rank of message

status(MPI_ERROR) – error code

Length of received message: MPI_Get_count()
Int MPI_Get_count(MPI_Status *status, MPI_Datatype datatype, int *count)

MPI_GET_COUNT(STATUS,DATATYPE,COUNT,IERROR)
integer STATUS(MPI_STATUS_SIZE),DATATYPE,COUNT,IERROR
MPI_Status status;
int count;
…
MPI_Recv(message,256,MPI_CHAR,0,99,MPI_COMM_WORLD,&status);
MPI_Get_count(&status, MPI_CHAR, &count); // count contains actual length