Implements the message passing interface (MPI) for high-performance computing (HPC). More...

#include <mpiWorld.h>

Inheritance diagram for MPIWorld:

Public Member Functions
	MPIWorld ()
	Initializes the MPI environment and finds our rank in the MPI world.
	~MPIWorld ()
int	size () const
int	rank () const
bool	isFirstInRank () const
bool	isLastInRank () const
void	exchangeBoundaries (void left_ghost, void left_edge, void right_edge, void right_ghost, int edge_size)
	Send our edges (they become other's ghosts) and receive other's edges (they become our ghosts).
void	waitExchange ()
	Wait the other ranks to have received our edges.
Private Attributes
int	worldSize
	The size of the MPI world.
int	myRank
	Our rank in the MPI world.
int	leftRank
	The rank of the left neighbor.
int	rightRank
	The rank of the right neighbor.
MPI_Request	waitLeft
	Outstanding MPI_Isend request to the left rank.
MPI_Request	waitRight
	Outstanding MPI_Isend request to the right rank.

Detailed Description

Implements the message passing interface (MPI) for high-performance computing (HPC).

In the MPI environment, the calculations on the grid are split among the ranks so the edges are exchanged between each two neighbors in the following way:

        .---- left -----.-------.---- right ----.
        | ghost | edge  |       | edge  | ghost |
        +-------+-----------------------+-------+
   ...  |       |     grid chunk i      |       |
        +-------+-----------------------+-------+------------+-------+
                                |       |   grid chunk i+1   |       |  ...
                                +-------+--------------------+-------+
                                :     left      :
                                : ghost | edge  :
                                `-------v-------´
                         exchanged between two neighbors

The overlapping edges of one rank become other's ghosts (of the same size). The ghost of the left-most rank is determined from the inner boundary conditions and the ghost of the right-most rank is determined from the outer boundary conditions.

When compiling, use mpic++ with -D_USEMPI. To run, use mpiexec.

Warning:: The maximal slicing is not compliant with MPI since the boundary value problem for the slicing differential equation requires access to the whole grid!

Definition at line 43 of file mpiWorld.h.

Constructor & Destructor Documentation

MPIWorld::MPIWorld ( ) [inline]

Initializes the MPI environment and finds our rank in the MPI world.

Definition at line 56 of file mpiWorld.h.

References isFirstInRank(), isLastInRank(), leftRank, myRank, rightRank, waitLeft, waitRight, and worldSize.

    {
        // Initialize the MPI environment
        //
        MPI_Init( NULL, NULL );

        // Find out the rank and the size of the MPI
        //
        MPI_Comm_rank( MPI_COMM_WORLD, &myRank );
        MPI_Comm_size( MPI_COMM_WORLD, &worldSize );

        leftRank  = isFirstInRank() ? -1 : myRank - 1;
        rightRank = isLastInRank()  ? -1 : myRank + 1;

        waitLeft = waitRight = MPI_REQUEST_NULL;

        // std::printf( "my rank %d, left %d, right %d\n", myRank, leftRank, rightRank );
    }

MPIWorld::~MPIWorld ( ) [inline]

Definition at line 75 of file mpiWorld.h.

    {
         MPI_Finalize ();
    }

Member Function Documentation

void MPIWorld::exchangeBoundaries	(	void *	left_ghost,
		void *	left_edge,
		void *	right_edge,
		void *	right_ghost,
		int	edge_size
	)		`[inline]`

Send our edges (they become other's ghosts) and receive other's edges (they become our ghosts).

Parameters:

left_ghost	Where to put the right edge of the left neighbor
left_edge	Pointer to the left edge of our data
right_edge	Pointer to the right edge of our data
right_ghost	Where to put the left edge of the right neighbor
edge_size	The edge (or ghost) size

Definition at line 100 of file mpiWorld.h.

References leftRank, rightRank, waitLeft, and waitRight.

Referenced by MPIWorld_sanityCheck().

    {
        waitLeft = waitRight = MPI_REQUEST_NULL;

        MPI_Datatype data_type = sizeof(Real) == sizeof(double)
                               ? MPI_DOUBLE : MPI_LONG_DOUBLE;

        // Send our edges (they become other's ghosts)

        if ( leftRank >= 0 ) {
            MPI_Isend( left_edge, edge_size, data_type, leftRank, 0,
                       MPI_COMM_WORLD, &waitLeft );
        }
        if ( rightRank >= 0 ) {
            MPI_Isend( right_edge, edge_size, data_type, rightRank, 0,
                       MPI_COMM_WORLD, &waitRight );
        }

        // Receive other's edges (they become our ghosts)
        //
        if ( leftRank >= 0 ) {
            MPI_Recv( left_ghost, edge_size, data_type, leftRank, 0,
                      MPI_COMM_WORLD, MPI_STATUS_IGNORE );
        }
        if ( rightRank >= 0 ) {
            MPI_Recv( right_ghost, edge_size, data_type, rightRank, 0,
                      MPI_COMM_WORLD, MPI_STATUS_IGNORE );
        }
    }

bool MPIWorld::isFirstInRank ( ) const [inline]

Definition at line 88 of file mpiWorld.h.

References myRank.

Referenced by MoLIntegrator::integStep_Begin(), and MPIWorld().

                                {
        return myRank == 0;
    }

bool MPIWorld::isLastInRank ( ) const [inline]

Definition at line 92 of file mpiWorld.h.

References myRank, and worldSize.

Referenced by MoLIntegrator::integStep_Begin(), and MPIWorld().

                               {
        return myRank == worldSize - 1;
    }

int MPIWorld::rank ( ) const [inline]

Definition at line 84 of file mpiWorld.h.

References myRank.

Referenced by UniformGrid::mpiRank(), and MPIWorld_sanityCheck().

                      {
        return myRank;
    }

int MPIWorld::size ( ) const [inline]

Definition at line 80 of file mpiWorld.h.

References worldSize.

Referenced by UniformGrid::mpiSize(), MPIWorld_sanityCheck(), and UniformGrid::UniformGrid().

                      {
        return worldSize;
    }

void MPIWorld::waitExchange ( ) [inline]

Wait the other ranks to have received our edges.

Definition at line 138 of file mpiWorld.h.

References waitLeft, and waitRight.

Referenced by MoLIntegrator::integStep_End(), and MPIWorld_sanityCheck().

    {
        if ( waitLeft != NULL ) {
            MPI_Status status;
            MPI_Wait( &waitLeft, &status );
        }
        if ( waitRight != NULL ) {
            MPI_Status status;
            MPI_Wait( &waitRight, &status );
        }
    }