MPI (Message Passing Interface)¶

MPI is the standard for running parallel applications across multiple nodes. Our clusters provide two MPI implementations through the module system: OpenMPI (in the foss toolchain) and Intel MPI (in the intel toolchain).

Available MPI implementations¶

Toolchain	MPI implementation	Load with
`foss`	OpenMPI	`module load foss/2024a`
`intel`	Intel MPI	`module load intel/2024a`

Both implementations are compiled with SLURM support on Atlas EDR and Hyperion, which means srun can launch MPI processes directly without needing mpirun.

To see available versions:

module spider foss
module spider intel

Compiling MPI programs¶

MPI provides compiler wrappers that link the necessary MPI libraries automatically. Do not try to link MPI libraries manually - always use the wrappers.

Compiler wrappers¶

Language	OpenMPI (foss)	Intel MPI Classic	Intel MPI oneAPI
C	`mpicc`	`mpiicc`	`mpiicx`
C++	`mpicxx`	`mpiicpc`	`mpiicpx`
Fortran	`mpif90`	`mpiifort`	`mpiifx`

Classic vs oneAPI wrappers

Intel is deprecating the classic wrappers (mpiicc, mpiicpc, mpiifort). For new projects, use the oneAPI wrappers (mpiicx, mpiicpx, mpiifx).

Compilation examples¶

foss (OpenMPI)intel (Intel MPI + oneAPI)

module load foss/2024a
mpicc -O2 -o myprogram myprogram.c
mpicxx -O2 -o myprogram myprogram.cpp
mpif90 -O2 -o myprogram myprogram.f90

module load intel/2024a
mpiicx -O2 -o myprogram myprogram.c
mpiicpx -O2 -o myprogram myprogram.cpp
mpiifx -O2 -o myprogram myprogram.f90

You can pass the same optimization flags described in the Compilers page. The wrappers forward them to the underlying compiler. To see exactly what a wrapper does:

mpicc --show    # OpenMPI
mpiicx -show    # Intel MPI

Running MPI jobs with SLURM¶

Use srun to launch MPI programs inside SLURM jobs. SLURM automatically handles process distribution across nodes based on your resource request.

Single-node job¶

mpi_job.slurm

#!/bin/bash
#SBATCH --job-name=my_mpi_job
#SBATCH --partition=general
#SBATCH --nodes=1
#SBATCH --ntasks-per-node=40
#SBATCH --time=01:00:00
#SBATCH --output=%x.%j.out
#SBATCH --error=%x.%j.err

module load intel/2024a

srun ./myprogram

Multi-node job¶

mpi_job.slurm

#!/bin/bash
#SBATCH --job-name=my_mpi_job
#SBATCH --partition=general
#SBATCH --nodes=4
#SBATCH --ntasks-per-node=40
#SBATCH --time=04:00:00
#SBATCH --output=%x.%j.out
#SBATCH --error=%x.%j.err

module load foss/2024a

srun ./myprogram

This launches 160 MPI processes (4 nodes x 40 tasks per node). SLURM distributes them automatically.

Why srun instead of mpirun?

On Atlas EDR and Hyperion, both OpenMPI and Intel MPI are compiled with SLURM support. Using srun provides better integration: SLURM controls process placement, handles signals correctly, and accounts for resource usage accurately. You do not need to pass -np or hostfiles - SLURM already knows this from your #SBATCH directives.

Atlas FDR

Atlas FDR does not have full SLURM-MPI integration. On this cluster, use mpirun instead:

mpirun -np ${SLURM_NTASKS} ./myprogram

Hybrid MPI + OpenMP jobs¶

Many applications use MPI across nodes and OpenMP threads within each node. In this case, you need to request the right combination of tasks and CPUs per task:

hybrid_job.slurm

#!/bin/bash
#SBATCH --job-name=hybrid_job
#SBATCH --partition=general
#SBATCH --nodes=2
#SBATCH --ntasks-per-node=4
#SBATCH --cpus-per-task=10
#SBATCH --time=02:00:00
#SBATCH --output=%x.%j.out
#SBATCH --error=%x.%j.err

module load foss/2024a

export OMP_NUM_THREADS=${SLURM_CPUS_PER_TASK}

srun ./myprogram

This runs 4 MPI processes per node, each with 10 OpenMP threads, on 2 nodes (8 MPI processes total, 80 cores total).

Tip

Set OMP_NUM_THREADS to match --cpus-per-task. If you forget, OpenMP may default to using all cores on the node, causing oversubscription.

MPI for Python (mpi4py)¶

If you need MPI from Python, mpi4py is available both as a system module and through conda. Since mpi4py must be compiled against the cluster's MPI library to work correctly, how you install it depends on which Python approach you are using.

Using system modules¶

If your workflow already uses system modules (e.g. SciPy-bundle, Python-bundle-PyPI), load mpi4py as a separate module:

module spider mpi4py
module load mpi4py/3.1.5-gompi-2024a

The module name includes the toolchain it was built against (e.g. gompi for GCC + OpenMPI, iimpi for Intel). Make sure it matches the toolchain you are using for the rest of your workflow. You can extend this setup with a venv if you need additional packages on top (see Python: venv with system modules).

Using Miniforge3 (conda)¶

If you manage your own conda environments, install mpi4py from conda-forge:

mamba install -c conda-forge mpi4py

Warning

Do not install mpi4py with pip install mpi4py on the cluster. Pip will try to compile it against whatever MPI it finds (or download a bundled one), which often results in a broken or suboptimal installation. Use the system module or conda instead.

Example SLURM job¶

mpi4py_job.slurm

#!/bin/bash
#SBATCH --job-name=mpi4py_job
#SBATCH --partition=general
#SBATCH --nodes=1
#SBATCH --ntasks-per-node=4
#SBATCH --time=00:30:00
#SBATCH --output=%x.%j.out
#SBATCH --error=%x.%j.err

module load mpi4py/3.1.5-gompi-2024a

srun python my_mpi_script.py