Magus02/OpenMP Usage - Revision history

Deepak: Created page with "{{DISPLAYTITLE:OpenMP Usage — Magus02 HPC}}

2026-04-20T06:43:09Z

Created page with "{{DISPLAYTITLE:OpenMP Usage — Magus02 HPC}} <div style="background:#0d1b2a;border:1px solid #1e3a5a;border-radius:8px;padding:12px 18px;margin-bottom:20px;font-size:13px;fon..."

New page
{{DISPLAYTITLE:OpenMP Usage — Magus02 HPC}}
<div style="background:#0d1b2a;border:1px solid #1e3a5a;border-radius:8px;padding:12px 18px;margin-bottom:20px;font-size:13px;font-family:monospace;color:#5ba3d9;">
📍 [[Magus02/Knowledge_Base|Magus02 Knowledge Base]] → HPC User → OpenMP Usage
</div>

OpenMP (Open Multi-Processing) is the standard API for '''shared-memory parallel programming''' on multi-core processors. It uses compiler directives (pragmas) to parallelize loops and code regions across multiple threads on a single node.

TOC

== What is OpenMP? ==

OpenMP allows a program to use multiple CPU cores within a single compute node by spawning '''threads''' that work simultaneously on different parts of the data. Unlike [[Magus02/MPI_Usage|MPI]], OpenMP threads share the same memory space — no message passing is needed.

On Magus02:
* Each AMD EPYC 7763 Compute Node has '''128 cores''' — ideal for large OpenMP thread counts
* Each AMD EPYC 7543 High Memory Node has '''64 cores'''
* OpenMP does '''not''' scale across multiple nodes — use MPI or Hybrid jobs for that

<div style="background:#0d1a2a;border-left:4px solid #00c9ff;border-radius:6px;padding:14px 18px;margin:16px 0;font-size:14px;color:#94a8be;">
💡 No additional module is needed for OpenMP itself — OpenMP support is built into the compiler. You simply need to load your compiler module (Intel or GCC) and add the appropriate flag.
</div>

== Compiling OpenMP Code ==

Add the OpenMP flag for your compiler when compiling:

{| class="wikitable" style="width:100%;"
|-
! style="background:#0d1b2a;color:#94a8be;" | Compiler
! style="background:#0d1b2a;color:#94a8be;" | Flag
! style="background:#0d1b2a;color:#94a8be;" | Example Command
|-
| GNU (<code>gcc</code> / <code>gfortran</code>) || <code>-fopenmp</code> || <code>gcc -fopenmp -O2 mycode.c -o myprogram</code>
|-
| Intel (<code>icc</code> / <code>ifort</code>) || <code>-qopenmp</code> || <code>icc -qopenmp -O2 mycode.c -o myprogram</code>
|-
| Clang || <code>-fopenmp</code> || <code>clang -fopenmp -O2 mycode.c -o myprogram</code>
|-
| Oracle Studio || <code>-xopenmp</code> || <code>cc -xopenmp -O2 mycode.c -o myprogram</code>
|-
| NAG Fortran || <code>-openmp</code> || <code>nagfor -openmp -O2 mycode.f90 -o myprogram</code>
|}

Without the OpenMP flag, all <code>#pragma omp</code> directives in your code are silently ignored and the program runs serially.

=== Example: Compiling with Intel compiler ===

<pre>
$ module purge
$ module load intel/oneapi

# Compile an OpenMP C program:
$ icc -qopenmp -O2 -march=native omp_program.c -o omp_program

# Compile an OpenMP Fortran program:
$ ifort -qopenmp -O2 -march=native omp_program.f90 -o omp_program
</pre>

=== Example: Compiling with GCC ===

<pre>
$ module purge
$ module load gcc/11.2.0

$ gcc -fopenmp -O2 -march=native omp_program.c -o omp_program
</pre>

== Running an OpenMP Program ==

=== Setting OMP_NUM_THREADS ===

'''Always set <code>OMP_NUM_THREADS</code>''' before running an OpenMP program. If unset, behaviour depends on the system default — sometimes it defaults to 1 (serial), sometimes to all available cores (potentially overloading the node).

Set it for a single execution:

<pre>
$ OMP_NUM_THREADS=32 ./omp_program
</pre>

Or export it to the environment for the session:

<pre>
$ export OMP_NUM_THREADS=64
$ ./omp_program
</pre>

In a Slurm jobscript, tie it to the allocated cores:

<pre>
export OMP_NUM_THREADS=$SLURM_CPUS_PER_TASK
</pre>

=== Complete Slurm Jobscript for OpenMP ===

<pre>
#!/bin/bash
#SBATCH --job-name=openmp_run
#SBATCH --partition=medium
#SBATCH --nodes=1
#SBATCH --ntasks=1
#SBATCH --cpus-per-task=64
#SBATCH --mem=256G
#SBATCH --time=06:00:00
#SBATCH --output=omp_%j.out
#SBATCH --error=omp_%j.err

module purge
module load intel/oneapi

# Set threads to match allocated cores:
export OMP_NUM_THREADS=$SLURM_CPUS_PER_TASK

# Thread placement for NUMA-awareness:
export OMP_PLACES=cores
export OMP_PROC_BIND=close

./omp_program input.dat
</pre>

== Thread Pinning ==

On Magus02's AMD EPYC nodes, each node has '''multiple NUMA domains''' (memory regions local to groups of cores). Pinning OpenMP threads to specific cores prevents the operating system from migrating threads between cores or NUMA domains, improving memory locality and performance.

Key environment variables:

{| class="wikitable" style="width:100%;"
|-
! Variable !! Values !! Effect
|-
| <code>OMP_PLACES</code> || <code>cores</code>, <code>threads</code>, <code>sockets</code> || Defines the "places" threads can be bound to
|-
| <code>OMP_PROC_BIND</code> || <code>close</code>, <code>spread</code>, <code>master</code> || Strategy for distributing threads across places
|}

Recommended for Magus02 (maximises cache locality):

<pre>
export OMP_PLACES=cores
export OMP_PROC_BIND=close
</pre>

For jobs that benefit from spreading across NUMA domains:

<pre>
export OMP_PLACES=sockets
export OMP_PROC_BIND=spread
</pre>

See [[Magus02/NUMA|NUMA Architecture]] for deeper explanation of why this matters on multi-socket nodes.

== Verifying Parallelism ==

After launching, check that threads are actually running in parallel:

<pre>
# From a separate terminal while the job runs (replace NODE with actual node name):
$ ssh NODE
$ top
# You should see your program using well over 100% CPU (e.g. 6400% for 64 threads)
</pre>

Within your program, you can print the thread count:

<pre>
#include <omp.h>
#include <stdio.h>

int main() {
#pragma omp parallel
{
int tid = omp_get_thread_num();
int nthreads = omp_get_num_threads();
if (tid == 0)
printf("Running with %d threads\n", nthreads);
}
return 0;
}
</pre>

== References ==

* [[Magus02/NUMA|NUMA Architecture — Thread Pinning]]
* [[Magus02/Batch_Scheduler|Batch Scheduler — OpenMP Jobscript]]
* [[Magus02/MPI_Usage|MPI Usage — Distributed Parallelism]]
* [[Magus02/Compiler|Compilers on Magus02]]

----
<div style="font-family:monospace;font-size:11px;color:#484f58;text-align:right;">
[[Magus02/Knowledge_Base|↑ Back to Knowledge Base]] · Maintained by HPC Administration · wiki.snu.edu.in
</div>