NUMA Architecture — Magus02 HPC

📍 Magus02 Knowledge Base → HPC User → NUMA Architecture

NUMA (Non-Uniform Memory Access) is the memory architecture used in Magus02's compute nodes. Understanding it helps you write programs that achieve maximum memory bandwidth and avoid hidden performance bottlenecks.

What is NUMA?

A standard desktop computer has one CPU and one block of main memory — all memory accesses take roughly the same time. A NUMA system (like Magus02's nodes) has multiple CPU sockets, each with its own local block of memory, connected by a high-speed interconnect.

Key properties:

Every core on every socket can access all memory — it is still a shared-memory system
Accessing local memory (same socket) is fast
Accessing remote memory (different socket, via interconnect) is slower
The name "Non-Uniform Memory Access" reflects this difference in access latency

Magus02 Node Topology

Node Type	Processor	Sockets	Cores/Socket	Local RAM/Socket	NUMA Domains
Compute Node	AMD EPYC 7763	2	64	256 GB	2 (minimum; EPYC has up to 8 per chip)
High Memory Node	AMD EPYC 7543	2	32	512 GB	2+
GPU Node	Intel Xeon Platinum 8368	2	32	256 GB	2

💡 AMD EPYC specifics: AMD EPYC "Milan" (7763) and "Rome" (7543) processors use a chiplet design with up to 8 internal NUMA domains per physical socket (called CCD or "core complex die"). This means even a single-socket view has multiple memory latency tiers. Use numactl --hardware on a node to see the exact topology.

Check the NUMA topology of a node:

# From an interactive session on a compute node:
$ numactl --hardware
$ numactl --show
$ lstopo            # graphical topology (requires X11)

Why NUMA Matters for Your Code

Two performance pitfalls arise from being NUMA-unaware:

Pitfall 1: Thread Migration

The Linux operating system is allowed to move threads between CPU cores to balance load. On a NUMA system, if a thread gets migrated from Socket 0 to Socket 1, it is now separated from its data (which was allocated on Socket 0's local memory). Every memory access then goes over the slow interconnect.

Solution: Thread pinning — bind each thread to a fixed core so the OS cannot migrate it.

Pitfall 2: Wrong Data Placement (First-Touch Policy)

When memory is allocated in C/Fortran (e.g. with malloc or array declaration), no physical memory is assigned yet — only virtual addresses. The operating system allocates the actual physical memory pages the first time each page is written (the "first touch"). Pages are placed in the local memory of the socket where the first-touching thread runs.

If you initialise all your arrays in the main thread (on Socket 0) before spawning OpenMP threads, all memory lands on Socket 0. Threads on Socket 1 then always access remote memory — severely limiting bandwidth.

Solution: Initialise data in parallel, using the same thread distribution you will use for computation.

NUMA-Aware Array Initialisation (OpenMP)

// WRONG — all data initialised by main thread → lands on Socket 0:
for (int i = 0; i < N; i++) {
    a[i] = 0.0;
    b[i] = i;
}
#pragma omp parallel for
for (int i = 0; i < N; i++) {
    a[i] = a[i] + b[i];   // threads on Socket 1 hit remote memory!
}


// CORRECT — data initialised in parallel → distributed across sockets:
#pragma omp parallel for
for (int i = 0; i < N; i++) {
    a[i] = 0.0;            // first-touch by the thread that will compute it
    b[i] = i;
}
#pragma omp parallel for
for (int i = 0; i < N; i++) {
    a[i] = a[i] + b[i];   // each thread accesses its own local data
}

Thread Pinning on Magus02

OpenMP Pinning

Set these environment variables in your jobscript or before running your program:

# Pin threads to cores, pack them close together (good for cache reuse):
export OMP_PLACES=cores
export OMP_PROC_BIND=close

# Alternative: spread threads evenly across NUMA domains (good for bandwidth):
export OMP_PLACES=sockets
export OMP_PROC_BIND=spread

numactl — Manual Process Binding

For serial or MPI programs, use numactl to bind a process to specific NUMA nodes:

# Run program using only Socket 0's cores and memory:
$ numactl --cpunodebind=0 --membind=0 ./my_program

# Run on Socket 1 only:
$ numactl --cpunodebind=1 --membind=1 ./my_program

# Run using all cores but memory local to the running thread:
$ numactl --localalloc ./my_program

MPI Pinning

See MPI Usage — Process Binding for binding MPI ranks to specific NUMA domains using Open MPI and Intel MPI options.

Checking Memory Bandwidth

The performance difference between local and remote memory access can be significant (often 2–3×). You can benchmark it using the STREAM benchmark if available on the cluster:

$ module load intel/oneapi
$ icc -O3 -qopenmp -march=native stream.c -o stream
$ OMP_NUM_THREADS=64 OMP_PROC_BIND=close ./stream

References

↑ Back to Knowledge Base · Maintained by HPC Administration · wiki.snu.edu.in