Magus02/OpenMP Usage

📍 Magus02 Knowledge Base → HPC User → OpenMP Usage

OpenMP (Open Multi-Processing) is the standard API for shared-memory parallel programming on multi-core processors. It uses compiler directives (pragmas) to parallelize loops and code regions across multiple threads on a single node.

What is OpenMP?

OpenMP allows a program to use multiple CPU cores within a single compute node by spawning threads that work simultaneously on different parts of the data. Unlike MPI, OpenMP threads share the same memory space — no message passing is needed.

On Magus02:

• Each AMD EPYC 7763 Compute Node has 128 cores — ideal for large OpenMP thread counts
• Each AMD EPYC 7543 High Memory Node has 64 cores
• OpenMP does not scale across multiple nodes — use MPI or Hybrid jobs for that

💡 No additional module is needed for OpenMP itself — OpenMP support is built into the compiler. You simply need to load your compiler module (Intel or GCC) and add the appropriate flag.

Compiling OpenMP Code

Add the OpenMP flag for your compiler when compiling:

Compiler	Flag	Example Command
GNU (gcc / gfortran)	-fopenmp	gcc -fopenmp -O2 mycode.c -o myprogram
Intel (icc / ifort)	-qopenmp	icc -qopenmp -O2 mycode.c -o myprogram
Clang	-fopenmp	clang -fopenmp -O2 mycode.c -o myprogram
Oracle Studio	-xopenmp	cc -xopenmp -O2 mycode.c -o myprogram
NAG Fortran	-openmp	nagfor -openmp -O2 mycode.f90 -o myprogram

Without the OpenMP flag, all #pragma omp directives in your code are silently ignored and the program runs serially.

Example: Compiling with Intel compiler

$ module purge
$ module load intel/oneapi

# Compile an OpenMP C program:
$ icc -qopenmp -O2 -march=native omp_program.c -o omp_program

# Compile an OpenMP Fortran program:
$ ifort -qopenmp -O2 -march=native omp_program.f90 -o omp_program

Example: Compiling with GCC

$ module purge
$ module load gcc/11.2.0

$ gcc -fopenmp -O2 -march=native omp_program.c -o omp_program

Running an OpenMP Program

Setting OMP_NUM_THREADS

Always set OMP_NUM_THREADS before running an OpenMP program. If unset, behaviour depends on the system default — sometimes it defaults to 1 (serial), sometimes to all available cores (potentially overloading the node).

Set it for a single execution:

$ OMP_NUM_THREADS=32 ./omp_program

Or export it to the environment for the session:

$ export OMP_NUM_THREADS=64
$ ./omp_program

In a Slurm jobscript, tie it to the allocated cores:

export OMP_NUM_THREADS=$SLURM_CPUS_PER_TASK

Complete Slurm Jobscript for OpenMP

#!/bin/bash
#SBATCH --job-name=openmp_run
#SBATCH --partition=medium
#SBATCH --nodes=1
#SBATCH --ntasks=1
#SBATCH --cpus-per-task=64
#SBATCH --mem=256G
#SBATCH --time=06:00:00
#SBATCH --output=omp_%j.out
#SBATCH --error=omp_%j.err

module purge
module load intel/oneapi

# Set threads to match allocated cores:
export OMP_NUM_THREADS=$SLURM_CPUS_PER_TASK

# Thread placement for NUMA-awareness:
export OMP_PLACES=cores
export OMP_PROC_BIND=close

./omp_program input.dat

Thread Pinning

On Magus02's AMD EPYC nodes, each node has multiple NUMA domains (memory regions local to groups of cores). Pinning OpenMP threads to specific cores prevents the operating system from migrating threads between cores or NUMA domains, improving memory locality and performance.

Key environment variables:

Variable	Values	Effect
OMP_PLACES	cores, threads, sockets	Defines the "places" threads can be bound to
OMP_PROC_BIND	close, spread, master	Strategy for distributing threads across places

Recommended for Magus02 (maximises cache locality):

export OMP_PLACES=cores
export OMP_PROC_BIND=close

For jobs that benefit from spreading across NUMA domains:

export OMP_PLACES=sockets
export OMP_PROC_BIND=spread

See NUMA Architecture for deeper explanation of why this matters on multi-socket nodes.

Verifying Parallelism

After launching, check that threads are actually running in parallel:

# From a separate terminal while the job runs (replace NODE with actual node name):
$ ssh NODE
$ top
# You should see your program using well over 100% CPU (e.g. 6400% for 64 threads)

Within your program, you can print the thread count:

#include <omp.h>
#include <stdio.h>

int main() {
    #pragma omp parallel
    {
        int tid = omp_get_thread_num();
        int nthreads = omp_get_num_threads();
        if (tid == 0)
            printf("Running with %d threads\n", nthreads);
    }
    return 0;
}

References

• NUMA Architecture — Thread Pinning
• Batch Scheduler — OpenMP Jobscript
• MPI Usage — Distributed Parallelism
• Compilers on Magus02

---

↑ Back to Knowledge Base · Maintained by HPC Administration · wiki.snu.edu.in