Have a Question?
Parallelization using MPI
Why MPI?
While using MPI, parallelism is achieved through the distribution of tasks across multiple processes, each with its memory space. MPI Processes have their own memory space, and data sharing is achieved through explicit message passing. Therefore, we need to explicitly manage communication between processes using MPI functions. Programs need to be explicitly written for MPI and may require modification when moving to different architectures.
MPI vs OpenMP
MPI
- Distributed Memory Model
- Explicit message passing between processes.
- Typically used for parallel programming on clusters and distributed-memory architectures.
- Processes can run independently, each with its own memory space.
OpenMP
- Shared Memory Model
- Implicit sharing of data through shared memory.
- Suited for parallel programming on shared-memory architectures, like multi-core machines.
- Threads within a process share the same memory space
Step 1: Create the Python Script
Use a text editor to create a Python script named hello_mpi.py. This script will use MPI to parallelize the “Hello World” program.
from mpi4py import MPI
import os
# Definition of the main function:
def main():
# Intialize the communicator and the rank
comm=MPI.COMM_WORLD
rank=comm.Get_rank()
size=comm.Get_size()
# Get the node name:
node_name = MPI.Get_processor_name()
# Print node address and process rank:
print(f"Hello! from node: {node_name}, Rank: {rank}/{size}, Process ID: {os.getpid()}")
if __name__ == "__main__":
main()
Step 2: Create the Slurm Job Script
#!/bin/bash
#SBATCH --job-name=hello_mpi
#SBATCH --output=hello_MPI.out
#SBATCH --error=hello_MPI.err
#SBATCH --nodes=2
#SBATCH --ntasks-per-node=5
#SBATCH --cpus-per-task=1
#SBATCH --time=00:01:00
#SBATCH --qos=proj-xxx
# Load the desired MPI version
module load openmpi/4.1.5
# set the number of processes:
NUM_PROCESSES=10 # run on 2 nodes.
# Run the MPI job:
mpirun -np $NUM_PROCESSES python3.6 hello_MPI.py
This script requests a total of 2 nodes (--nodes=2). On each node, it requests 5 tasks (--ntasks-per-node=5). Additionally, each task is allocated 1 CPU core (--cpus-per-task=1).
Step 3: Submit and Monitor the Job
Submit the job using the sbatch command.
$ sbatch hello_mpi.sh
Step 4: Inspect the Output/Error
Once the job completes, view the output in the files specified in your Slurm job script (hello_MPI.out and hello_MPI.err in this example). Use the cat command to view the content. The output should look similar to:
$ cat hello_MPI.out
Hello! from node: compute-04, Rank: 0/10, Process ID: 1865343
Hello! from node: compute-04, Rank: 1/10, Process ID: 1865344
Hello! from node: compute-04, Rank: 2/10, Process ID: 1865345
Hello! from node: compute-04, Rank: 3/10, Process ID: 1865346
Hello! from node: compute-04, Rank: 4/10, Process ID: 1865347
Hello! from node: compute-05, Rank: 5/10, Process ID: 1492484
Hello! from node: compute-05, Rank: 6/10, Process ID: 1492485
Hello! from node: compute-05, Rank: 7/10, Process ID: 1492486
Hello! from node: compute-05, Rank: 8/10, Process ID: 1492487
Hello! from node: compute-05, Rank: 9/10, Process ID: 1492488
Congratulations! You have successfully run a Slurm job with MPI parallelization for a “Hello World” Python script on ARCC2 cluster. This example serves as a starting point for more complex parallel computing tasks using MPI and Python. Adjust the script and parameters based on your specific requirements.