Magus02/NUMA - Revision history

Deepak: Created page with "{{DISPLAYTITLE:NUMA Architecture — Magus02 HPC}}

2026-04-20T06:45:23Z

Created page with "{{DISPLAYTITLE:NUMA Architecture — Magus02 HPC}} <div style="background:#0d1b2a;border:1px solid #1e3a5a;border-radius:8px;padding:12px 18px;margin-bottom:20px;font-size:13p..."

New page
{{DISPLAYTITLE:NUMA Architecture — Magus02 HPC}}
<div style="background:#0d1b2a;border:1px solid #1e3a5a;border-radius:8px;padding:12px 18px;margin-bottom:20px;font-size:13px;font-family:monospace;color:#5ba3d9;">
📍 [[Magus02/Knowledge_Base|Magus02 Knowledge Base]] → HPC User → NUMA Architecture
</div>

NUMA (Non-Uniform Memory Access) is the memory architecture used in Magus02's compute nodes. Understanding it helps you write programs that achieve maximum memory bandwidth and avoid hidden performance bottlenecks.

TOC

== What is NUMA? ==

A standard desktop computer has one CPU and one block of main memory — all memory accesses take roughly the same time. A '''NUMA system''' (like Magus02's nodes) has '''multiple CPU sockets''', each with its own local block of memory, connected by a high-speed interconnect.

Key properties:
* Every core on every socket '''can access all memory''' — it is still a shared-memory system
* Accessing '''local memory''' (same socket) is '''fast'''
* Accessing '''remote memory''' (different socket, via interconnect) is '''slower'''
* The name "Non-Uniform Memory Access" reflects this difference in access latency

=== Magus02 Node Topology ===

{| class="wikitable" style="width:100%;"
|-
! style="background:#0d1b2a;color:#94a8be;" | Node Type
! style="background:#0d1b2a;color:#94a8be;" | Processor
! style="background:#0d1b2a;color:#94a8be;" | Sockets
! style="background:#0d1b2a;color:#94a8be;" | Cores/Socket
! style="background:#0d1b2a;color:#94a8be;" | Local RAM/Socket
! style="background:#0d1b2a;color:#94a8be;" | NUMA Domains
|-
| Compute Node || AMD EPYC 7763 || 2 || 64 || 256 GB || 2 (minimum; EPYC has up to 8 per chip)
|-
| High Memory Node || AMD EPYC 7543 || 2 || 32 || 512 GB || 2+
|-
| GPU Node || Intel Xeon Platinum 8368 || 2 || 32 || 256 GB || 2
|}

<div style="background:#0d1a2a;border-left:4px solid #00c9ff;border-radius:6px;padding:14px 18px;margin:16px 0;font-size:14px;color:#94a8be;">
💡 <strong>AMD EPYC specifics:</strong> AMD EPYC "Milan" (7763) and "Rome" (7543) processors use a chiplet design with up to 8 internal NUMA domains per physical socket (called CCD or "core complex die"). This means even a single-socket view has multiple memory latency tiers. Use <code>numactl --hardware</code> on a node to see the exact topology.
</div>

Check the NUMA topology of a node:

<pre>
# From an interactive session on a compute node:
$ numactl --hardware
$ numactl --show
$ lstopo # graphical topology (requires X11)
</pre>

== Why NUMA Matters for Your Code ==

Two performance pitfalls arise from being NUMA-unaware:

=== Pitfall 1: Thread Migration ===

The Linux operating system is allowed to move threads between CPU cores to balance load. On a NUMA system, if a thread gets migrated from Socket 0 to Socket 1, it is now separated from its data (which was allocated on Socket 0's local memory). Every memory access then goes over the slow interconnect.

'''Solution: Thread pinning''' — bind each thread to a fixed core so the OS cannot migrate it.

=== Pitfall 2: Wrong Data Placement (First-Touch Policy) ===

When memory is allocated in C/Fortran (e.g. with <code>malloc</code> or array declaration), no physical memory is assigned yet — only virtual addresses. The operating system allocates the actual physical memory pages the first time each page is ''written'' (the "first touch"). Pages are placed in the local memory of the socket where the first-touching thread runs.

If you initialise all your arrays in the main thread (on Socket 0) before spawning OpenMP threads, all memory lands on Socket 0. Threads on Socket 1 then always access remote memory — severely limiting bandwidth.

'''Solution: Initialise data in parallel''', using the same thread distribution you will use for computation.

=== NUMA-Aware Array Initialisation (OpenMP) ===

<pre>
// WRONG — all data initialised by main thread → lands on Socket 0:
for (int i = 0; i < N; i++) {
a[i] = 0.0;
b[i] = i;
}
#pragma omp parallel for
for (int i = 0; i < N; i++) {
a[i] = a[i] + b[i]; // threads on Socket 1 hit remote memory!
}

// CORRECT — data initialised in parallel → distributed across sockets:
#pragma omp parallel for
for (int i = 0; i < N; i++) {
a[i] = 0.0; // first-touch by the thread that will compute it
b[i] = i;
}
#pragma omp parallel for
for (int i = 0; i < N; i++) {
a[i] = a[i] + b[i]; // each thread accesses its own local data
}
</pre>

== Thread Pinning on Magus02 ==

=== OpenMP Pinning ===

Set these environment variables in your jobscript or before running your program:

<pre>
# Pin threads to cores, pack them close together (good for cache reuse):
export OMP_PLACES=cores
export OMP_PROC_BIND=close

# Alternative: spread threads evenly across NUMA domains (good for bandwidth):
export OMP_PLACES=sockets
export OMP_PROC_BIND=spread
</pre>

=== numactl — Manual Process Binding ===

For serial or MPI programs, use <code>numactl</code> to bind a process to specific NUMA nodes:

<pre>
# Run program using only Socket 0's cores and memory:
$ numactl --cpunodebind=0 --membind=0 ./my_program

# Run on Socket 1 only:
$ numactl --cpunodebind=1 --membind=1 ./my_program

# Run using all cores but memory local to the running thread:
$ numactl --localalloc ./my_program
</pre>

=== MPI Pinning ===

See [[Magus02/MPI_Usage#Process_Binding|MPI Usage — Process Binding]] for binding MPI ranks to specific NUMA domains using Open MPI and Intel MPI options.

== Checking Memory Bandwidth ==

The performance difference between local and remote memory access can be significant (often 2–3×). You can benchmark it using the STREAM benchmark if available on the cluster:

<pre>
$ module load intel/oneapi
$ icc -O3 -qopenmp -march=native stream.c -o stream
$ OMP_NUM_THREADS=64 OMP_PROC_BIND=close ./stream
</pre>

== References ==

* [https://software.intel.com/en-us/articles/optimizing-applications-for-numa Intel: Optimizing Applications for NUMA]
* [[Magus02/OpenMP_Usage|OpenMP Usage — Thread Pinning]]
* [[Magus02/MPI_Usage|MPI Usage — Process Binding]]
* [[Magus02/Batch_Scheduler|Batch Scheduler — Hybrid Jobs]]

----
<div style="font-family:monospace;font-size:11px;color:#484f58;text-align:right;">
[[Magus02/Knowledge_Base|↑ Back to Knowledge Base]] · Maintained by HPC Administration · wiki.snu.edu.in
</div>