MPI Usage — Magus02 HPC

📍 Magus02 Knowledge Base → HPC User → MPI Usage

MPI (Message Passing Interface) is the standard for distributed-memory parallel programming — running a program as multiple independent processes that communicate by passing messages, potentially across many compute nodes.

What is MPI?

MPI programs spawn multiple processes (as opposed to OpenMP's threads), each with its own separate memory space. Processes coordinate by explicitly sending and receiving messages. This allows MPI programs to scale across dozens or hundreds of nodes, making it the dominant model for large-scale HPC workloads.

On Magus02:

Two MPI libraries are available: Open MPI and Intel MPI
Never load both at the same time — they conflict
MPI is always loaded via the module system

⚠️ Critical: Load exactly one MPI module. Loading both mpi/openmpi and mpi/intelmpi simultaneously will cause conflicts and unpredictable failures. Use module purge at the start of every jobscript.

Loading MPI Modules

Open MPI

$ module purge
$ module load gcc/11.2.0
$ module load mpi/openmpi

Intel MPI

$ module purge
$ module load intel/oneapi
$ module load mpi/intelmpi

Compiling MPI Code

MPI provides compiler wrappers that automatically include the correct headers and libraries. Always use these wrappers instead of calling gcc or icc directly.

Open MPI Compiler Wrappers

Language	Wrapper	Example
C	`mpicc`	`mpicc -O2 myprog.c -o myprog`
C++	`mpicxx`	`mpicxx -O2 myprog.cpp -o myprog`
Fortran	`mpifort`	`mpifort -O2 myprog.f90 -o myprog`

Intel MPI Compiler Wrappers

Intel MPI ships two sets of wrappers — one for GCC and one for Intel compilers. Use the right set for your code.

Underlying Compiler	C	C++	Fortran
GCC-based	`mpicc`	`mpicxx`	`mpifort`
Intel-based	`mpiicc`	`mpiicpc`	`mpiifort`

Example with Intel compilers:

$ module purge
$ module load intel/oneapi mpi/intelmpi

$ mpiicc  -O2 myprog.c   -o myprog   # C with Intel compiler
$ mpiicpc -O2 myprog.cpp -o myprog   # C++ with Intel compiler
$ mpiifort -O2 myprog.f90 -o myprog  # Fortran with Intel compiler

On most systems, environment variables are also available:

$ $MPICC  myprog.c   -O2 -o myprog
$ $MPICXX myprog.cpp -O2 -o myprog
$ $MPIFC  myprog.f90 -O2 -o myprog

Running MPI Programs

Basic Launch

# General form:
$ mpirun -n <num_processes> [options] ./my_program [program_args]

# Example: run with 256 MPI processes:
$ mpirun -n 256 ./my_mpi_program input.dat

Note: mpirun and mpiexec are synonymous in Open MPI. In Intel MPI it is mpiexec.hydra and mpirun.

Key mpirun Options

Open MPI

Option	Description
`-n <N>` or `-np <N>`	Total number of MPI processes
`-npernode <N>`	Number of processes per node
`-npersocket <N>`	Number of processes per socket
`-wdir <dir>`	Change working directory before execution
`-x VAR`	Export environment variable to remote processes
`-output-filename <name>`	Redirect output to files named <name>.<rank>
`--help`	List all available options

Intel MPI

Option	Description
`-n <N>`	Total number of MPI processes
`-ppn <N>`	Processes per node (set `I_MPI_JOB_RESPECT_PROCESS_PLACEMENT=off` if needed)
`-wdir <dir>`	Change working directory
`-outfile-pattern <name>`	Redirect stdout to file
`-g`	Enable debug symbols
`-O2` / `-O3`	Optimization level

Complete Slurm Jobscript — MPI

#!/bin/bash
#SBATCH --job-name=mpi_vasp
#SBATCH --partition=medium
#SBATCH --nodes=4
#SBATCH --ntasks-per-node=64
#SBATCH --mem=512G
#SBATCH --time=7-00:00:00
#SBATCH --output=vasp_%j.out
#SBATCH --error=vasp_%j.err
#SBATCH --mail-type=END,FAIL
#SBATCH [email protected]

module purge
module load intel/oneapi
module load mpi/intelmpi
module load mkl

# Run with total tasks = nodes × ntasks-per-node = 256:
mpirun -n $SLURM_NTASKS ./vasp_std

Process Binding

Binding MPI processes to specific CPU cores improves performance by ensuring each process stays close to its local memory (NUMA-awareness). Both Open MPI and Intel MPI support binding.

⚠️ Multi-job binding conflict: MPI binding is on by default in most implementations. If you run more than one MPI job on the same hardware simultaneously (e.g. in a non-exclusive allocation), all jobs assume they are the only job and bind themselves to the same cores — resulting in 1/N performance while other cores sit idle.

Open MPI — Binding Options

# Bind one process per core:
$ mpirun -n 64 --bind-to core ./my_mpi_program

# Bind one process per socket:
$ mpirun -n 8 --bind-to socket ./my_mpi_program

# Report binding (for verification):
$ mpirun -n 64 --bind-to core --report-bindings ./my_mpi_program 2>&1 | head -20

Intel MPI — Binding Options

# Bind to cores:
$ export I_MPI_PIN=1
$ export I_MPI_PIN_DOMAIN=core
$ mpirun -n 64 ./my_mpi_program

# Check binding:
$ export I_MPI_DEBUG=4
$ mpirun -n 4 ./my_mpi_program

References

↑ Back to Knowledge Base · Maintained by HPC Administration · wiki.snu.edu.in