OpenMP (Open Multi-Processing)¶

OpenMP enables shared-memory parallelism within a single node using compiler directives and environment variables. It is supported by all compiler suites available on our clusters (GCC, Intel, NVIDIA HPC SDK).

Compiling with OpenMP¶

Add the appropriate flag when compiling:

Compiler suite	Flag	Example
GCC	`-fopenmp`	`gcc -fopenmp -O2 -o myprog myprog.c`
Intel oneAPI	`-qopenmp`	`icx -qopenmp -O2 -o myprog myprog.c`
Intel Classic	`-qopenmp`	`icc -qopenmp -O2 -o myprog myprog.c`
NVIDIA HPC SDK	`-mp`	`nvc -mp -O2 -o myprog myprog.c`

GCCIntel oneAPINVIDIA HPC SDK (Hyperion)

module load GCC/13.3.0
gcc -fopenmp -O2 -o myprog myprog.c
g++ -fopenmp -O2 -o myprog myprog.cpp
gfortran -fopenmp -O2 -o myprog myprog.f90

module load intel-compilers/2024.2.1
icx -qopenmp -O2 -o myprog myprog.c
icpx -qopenmp -O2 -o myprog myprog.cpp
ifx -qopenmp -O2 -o myprog myprog.f90

module load NVHPC/23.1-CUDA-12.1.1
nvc -mp -O2 -o myprog myprog.c
nvc++ -mp -O2 -o myprog myprog.cpp
nvfortran -mp -O2 -o myprog myprog.f90

For optimization flags specific to each cluster's hardware, see Compilers.

Running OpenMP jobs with SLURM¶

The key SLURM parameter for OpenMP is --cpus-per-task, which determines how many cores are available to your threads. Always set OMP_NUM_THREADS to match.

openmp_job.slurm

#!/bin/bash
#SBATCH --job-name=openmp_job
#SBATCH --partition=general
#SBATCH --nodes=1
#SBATCH --ntasks=1
#SBATCH --cpus-per-task=16
#SBATCH --time=02:00:00
#SBATCH --output=%x.%j.out
#SBATCH --error=%x.%j.err

module load GCC/13.3.0

export OMP_NUM_THREADS=${SLURM_CPUS_PER_TASK}

./myprog

Tip

Always use OMP_NUM_THREADS=${SLURM_CPUS_PER_TASK} rather than hardcoding a number. This ensures your program uses exactly the resources SLURM allocated, and makes your script portable across different resource requests.

Thread affinity and binding¶

By default, the operating system is free to migrate threads between cores, which can hurt performance. Pinning threads to specific cores avoids this.

Recommended settings¶

For most workloads on our clusters:

export OMP_PLACES=cores
export OMP_PROC_BIND=close

OMP_PLACES=cores assigns one thread per physical core.
OMP_PROC_BIND=close places threads near each other, which is beneficial when threads share data.

If your application benefits from spreading threads across sockets (e.g. to maximize memory bandwidth):

export OMP_PLACES=cores
export OMP_PROC_BIND=spread

Full SLURM example with binding¶

openmp_bound.slurm

#!/bin/bash
#SBATCH --job-name=openmp_bound
#SBATCH --partition=general
#SBATCH --nodes=1
#SBATCH --ntasks=1
#SBATCH --cpus-per-task=16
#SBATCH --time=02:00:00
#SBATCH --output=%x.%j.out
#SBATCH --error=%x.%j.err

module load GCC/13.3.0

export OMP_NUM_THREADS=${SLURM_CPUS_PER_TASK}
export OMP_PLACES=cores
export OMP_PROC_BIND=close

./myprog

Common environment variables¶

Variable	Purpose	Typical value
`OMP_NUM_THREADS`	Number of threads	`${SLURM_CPUS_PER_TASK}`
`OMP_PLACES`	Where to pin threads	`cores`
`OMP_PROC_BIND`	How to distribute threads	`close` or `spread`
`OMP_STACKSIZE`	Stack size per thread	`256M` (increase if segfaults)
`OMP_SCHEDULE`	Loop scheduling policy	`dynamic`, `static`, `guided`

Stack size

The default stack size per thread is often too small for scientific codes with large local arrays (particularly in Fortran). If your program crashes with a segmentation fault inside a parallel region, try increasing OMP_STACKSIZE:

export OMP_STACKSIZE=256M

Hybrid MPI + OpenMP¶

OpenMP is frequently combined with MPI for multi-node parallelism. See MPI: Hybrid MPI + OpenMP jobs for examples and recommended SLURM settings.