Magus02/MPI Usage - Revision history

Deepak: Created page with "{{DISPLAYTITLE:MPI Usage — Magus02 HPC}}

2026-04-20T06:44:11Z

Created page with "{{DISPLAYTITLE:MPI Usage — Magus02 HPC}} <div style="background:#0d1b2a;border:1px solid #1e3a5a;border-radius:8px;padding:12px 18px;margin-bottom:20px;font-size:13px;font-f..."

New page
{{DISPLAYTITLE:MPI Usage — Magus02 HPC}}
<div style="background:#0d1b2a;border:1px solid #1e3a5a;border-radius:8px;padding:12px 18px;margin-bottom:20px;font-size:13px;font-family:monospace;color:#5ba3d9;">
📍 [[Magus02/Knowledge_Base|Magus02 Knowledge Base]] → HPC User → MPI Usage
</div>

MPI (Message Passing Interface) is the standard for '''distributed-memory parallel programming''' — running a program as multiple independent processes that communicate by passing messages, potentially across many compute nodes.

TOC

== What is MPI? ==

MPI programs spawn multiple '''processes''' (as opposed to OpenMP's threads), each with its own separate memory space. Processes coordinate by explicitly sending and receiving messages. This allows MPI programs to scale across dozens or hundreds of nodes, making it the dominant model for large-scale HPC workloads.

On Magus02:
* Two MPI libraries are available: '''Open MPI''' and '''Intel MPI'''
* Never load both at the same time — they conflict
* MPI is always loaded via the [[Magus02/Modules|module system]]

<div style="background:#2a1a00;border-left:4px solid #f0c040;border-radius:6px;padding:14px 18px;margin:16px 0;font-size:14px;color:#d4b870;">
⚠️ <strong>Critical:</strong> Load exactly one MPI module. Loading both <code>mpi/openmpi</code> and <code>mpi/intelmpi</code> simultaneously will cause conflicts and unpredictable failures. Use <code>module purge</code> at the start of every jobscript.
</div>

== Loading MPI Modules ==

=== Open MPI ===

<pre>
$ module purge
$ module load gcc/11.2.0
$ module load mpi/openmpi
</pre>

=== Intel MPI ===

<pre>
$ module purge
$ module load intel/oneapi
$ module load mpi/intelmpi
</pre>

== Compiling MPI Code ==

MPI provides '''compiler wrappers''' that automatically include the correct headers and libraries. Always use these wrappers instead of calling <code>gcc</code> or <code>icc</code> directly.

=== Open MPI Compiler Wrappers ===

{| class="wikitable" style="width:100%;"
|-
! style="background:#0d1b2a;color:#94a8be;" | Language
! style="background:#0d1b2a;color:#94a8be;" | Wrapper
! style="background:#0d1b2a;color:#94a8be;" | Example
|-
| C || <code>mpicc</code> || <code>mpicc -O2 myprog.c -o myprog</code>
|-
| C++ || <code>mpicxx</code> || <code>mpicxx -O2 myprog.cpp -o myprog</code>
|-
| Fortran || <code>mpifort</code> || <code>mpifort -O2 myprog.f90 -o myprog</code>
|}

=== Intel MPI Compiler Wrappers ===

Intel MPI ships '''two sets''' of wrappers — one for GCC and one for Intel compilers. Use the right set for your code.

{| class="wikitable" style="width:100%;"
|-
! style="background:#0d1b2a;color:#94a8be;" | Underlying Compiler
! style="background:#0d1b2a;color:#94a8be;" | C
! style="background:#0d1b2a;color:#94a8be;" | C++
! style="background:#0d1b2a;color:#94a8be;" | Fortran
|-
| GCC-based || <code>mpicc</code> || <code>mpicxx</code> || <code>mpifort</code>
|-
| Intel-based || <code>mpiicc</code> || <code>mpiicpc</code> || <code>mpiifort</code>
|}

Example with Intel compilers:

<pre>
$ module purge
$ module load intel/oneapi mpi/intelmpi

$ mpiicc -O2 myprog.c -o myprog # C with Intel compiler
$ mpiicpc -O2 myprog.cpp -o myprog # C++ with Intel compiler
$ mpiifort -O2 myprog.f90 -o myprog # Fortran with Intel compiler
</pre>

On most systems, environment variables are also available:

<pre>
$ $MPICC myprog.c -O2 -o myprog
$ $MPICXX myprog.cpp -O2 -o myprog
$ $MPIFC myprog.f90 -O2 -o myprog
</pre>

== Running MPI Programs ==

=== Basic Launch ===

<pre>
# General form:
$ mpirun -n <num_processes> [options] ./my_program [program_args]

# Example: run with 256 MPI processes:
$ mpirun -n 256 ./my_mpi_program input.dat
</pre>

Note: <code>mpirun</code> and <code>mpiexec</code> are synonymous in Open MPI. In Intel MPI it is <code>mpiexec.hydra</code> and <code>mpirun</code>.

=== Key mpirun Options ===

==== Open MPI ====

{| class="wikitable" style="width:100%;"
|-
! style="background:#0d1b2a;color:#94a8be;" | Option
! style="background:#0d1b2a;color:#94a8be;" | Description
|-
| <code>-n <N></code> or <code>-np <N></code> || Total number of MPI processes
|-
| <code>-npernode <N></code> || Number of processes per node
|-
| <code>-npersocket <N></code> || Number of processes per socket
|-
| <code>-wdir <dir></code> || Change working directory before execution
|-
| <code>-x VAR</code> || Export environment variable to remote processes
|-
| <code>-output-filename <name></code> || Redirect output to files named <name>.<rank>
|-
| <code>--help</code> || List all available options
|}

==== Intel MPI ====

{| class="wikitable" style="width:100%;"
|-
! style="background:#0d1b2a;color:#94a8be;" | Option
! style="background:#0d1b2a;color:#94a8be;" | Description
|-
| <code>-n <N></code> || Total number of MPI processes
|-
| <code>-ppn <N></code> || Processes per node (set <code>I_MPI_JOB_RESPECT_PROCESS_PLACEMENT=off</code> if needed)
|-
| <code>-wdir <dir></code> || Change working directory
|-
| <code>-outfile-pattern <name></code> || Redirect stdout to file
|-
| <code>-g</code> || Enable debug symbols
|-
| <code>-O2</code> / <code>-O3</code> || Optimization level
|}

== Complete Slurm Jobscript — MPI ==

<pre>
#!/bin/bash
#SBATCH --job-name=mpi_vasp
#SBATCH --partition=medium
#SBATCH --nodes=4
#SBATCH --ntasks-per-node=64
#SBATCH --mem=512G
#SBATCH --time=7-00:00:00
#SBATCH --output=vasp_%j.out
#SBATCH --error=vasp_%j.err
#SBATCH --mail-type=END,FAIL
#SBATCH --mail-user=your_email@snu.edu.in

module purge
module load intel/oneapi
module load mpi/intelmpi
module load mkl

# Run with total tasks = nodes × ntasks-per-node = 256:
mpirun -n $SLURM_NTASKS ./vasp_std
</pre>

== Process Binding ==

Binding MPI processes to specific CPU cores improves performance by ensuring each process stays close to its local memory (NUMA-awareness). Both Open MPI and Intel MPI support binding.

<div style="background:#2a1a00;border-left:4px solid #f0c040;border-radius:6px;padding:14px 18px;margin:16px 0;font-size:14px;color:#d4b870;">
⚠️ <strong>Multi-job binding conflict:</strong> MPI binding is <strong>on by default</strong> in most implementations. If you run more than one MPI job on the same hardware simultaneously (e.g. in a non-exclusive allocation), all jobs assume they are the only job and bind themselves to the same cores — resulting in <em>1/N</em> performance while other cores sit idle.
</div>

=== Open MPI — Binding Options ===

<pre>
# Bind one process per core:
$ mpirun -n 64 --bind-to core ./my_mpi_program

# Bind one process per socket:
$ mpirun -n 8 --bind-to socket ./my_mpi_program

# Report binding (for verification):
$ mpirun -n 64 --bind-to core --report-bindings ./my_mpi_program 2>&1 | head -20
</pre>

=== Intel MPI — Binding Options ===

<pre>
# Bind to cores:
$ export I_MPI_PIN=1
$ export I_MPI_PIN_DOMAIN=core
$ mpirun -n 64 ./my_mpi_program

# Check binding:
$ export I_MPI_DEBUG=4
$ mpirun -n 4 ./my_mpi_program
</pre>

== References ==

* [[Magus02/NUMA|NUMA Architecture — Memory Locality]]
* [[Magus02/Batch_Scheduler|Batch Scheduler — MPI Jobscript]]
* [[Magus02/OpenMP_Usage|OpenMP Usage — Hybrid Jobs]]
* [[Magus02/Compiler|Compilers on Magus02]]
* [https://www.open-mpi.org/doc/v4.0/man1/mpiexec.1.php Open MPI mpiexec man page]
* [https://software.intel.com/content/www/us/en/develop/documentation/mpi-developer-reference-linux Intel MPI Reference]

----
<div style="font-family:monospace;font-size:11px;color:#484f58;text-align:right;">
[[Magus02/Knowledge_Base|↑ Back to Knowledge Base]] · Maintained by HPC Administration · wiki.snu.edu.in
</div>