Apptainer: advanced topics¶

This page covers advanced container usage. For the basics (pulling, running, building, GPU, SLURM), see Apptainer.

Writable overlays¶

SIF images are read-only. If you need to install additional packages or modify files inside a container without rebuilding it from scratch, use a writable overlay. An overlay is an ext3 filesystem image that sits on top of the read-only SIF and captures all changes.

Creating an overlay¶

Create a 2 GB overlay file:

apptainer overlay create --size 2048 my_overlay.img

Using the overlay¶

Attach the overlay to your container with --overlay and make it writable with --writable:

apptainer shell --overlay my_overlay.img --writable ubuntu_latest.sif

Inside this shell you can install packages, create files, and modify the filesystem. All changes are stored in the overlay, not in the SIF:

Apptainer> apt-get update && apt-get install -y curl
Apptainer> pip install pandas scikit-learn
Apptainer> exit

The next time you use the same overlay, your installed packages are still there:

apptainer exec --overlay my_overlay.img ubuntu_latest.sif python -c "import pandas; print(pandas.__version__)"

When to use overlays¶

Scenario	Recommended approach
Testing or prototyping packages	Overlay (quick iteration, no rebuild)
Production image shared with others	Definition file + `apptainer build` (reproducible)
Adding data or config to a third-party image	Overlay (avoids modifying the original)
Large dependency trees that change often	Overlay during development, then bake into a def file

Overlays in batch jobs¶

Overlays work seamlessly with SLURM. Just add the --overlay flag:

overlay_job.slurm

#!/bin/bash
#SBATCH --qos=regular
#SBATCH --job-name=overlay-test
#SBATCH --ntasks=1
#SBATCH --mem=4gb
#SBATCH --time=01:00:00
#SBATCH --output=%x.%j.out
#SBATCH --error=%x.%j.err

module load Apptainer

apptainer exec \
    --overlay /scratch/$USER/containers/my_overlay.img \
    --bind /scratch/$USER:/scratch/$USER \
    /scratch/$USER/containers/ubuntu_latest.sif \
    python /scratch/$USER/my_analysis.py

Warning

Do not use the same overlay file from multiple jobs simultaneously. Concurrent writes to the same ext3 image will corrupt it. If you need parallel jobs, create a separate overlay copy for each one, or use the overlay in read-only mode by appending :ro:

--overlay my_overlay.img:ro

My overlay is full

Create a larger one and copy the contents:

apptainer overlay create --size 4096 my_overlay_new.img

Then reinstall your packages into the new overlay, or use dd / filesystem tools to resize the ext3 image.

MPI inside containers (single node)¶

You can run MPI applications inside a container on a single node. In this case, the MPI library lives entirely inside the container and communicates through shared memory, so there is no dependency on the host MPI or the InfiniBand interconnect.

Example: building an MPI container¶

First, create the C source file. It passes a token around a ring of MPI ranks; each rank adds its own number to the token, so with 4 ranks the final value is 42 + 1 + 2 + 3 = 48:

ring.c

#include <mpi.h>
#include <stdio.h>

int main(int argc, char **argv) {
    int rank, size, token;
    MPI_Init(&argc, &argv);
    MPI_Comm_rank(MPI_COMM_WORLD, &rank);
    MPI_Comm_size(MPI_COMM_WORLD, &size);
    if (rank == 0) {
        token = 42;
        printf("Rank 0 starts token = %d (sending to %d ranks)\n", token, size);
        MPI_Send(&token, 1, MPI_INT, 1, 0, MPI_COMM_WORLD);
        MPI_Recv(&token, 1, MPI_INT, size - 1, 0, MPI_COMM_WORLD, MPI_STATUS_IGNORE);
        printf("Rank 0 received token = %d (after full ring)\n", token);
    } else {
        MPI_Recv(&token, 1, MPI_INT, rank - 1, 0, MPI_COMM_WORLD, MPI_STATUS_IGNORE);
        token += rank;
        printf("Rank %d received token, added %d, sending token = %d\n", rank, rank, token);
        MPI_Send(&token, 1, MPI_INT, (rank + 1) % size, 0, MPI_COMM_WORLD);
    }
    MPI_Finalize();
    return 0;
}

Then the definition file, which copies ring.c into the container and compiles it during the build:

mpi_ring.def

Bootstrap: docker
From: ubuntu:24.04

%files
    ring.c /opt/ring.c

%post
    apt-get update && apt-get install -y --no-install-recommends \
        build-essential openmpi-bin libopenmpi-dev
    rm -rf /var/lib/apt/lists/*
    mpicc -o /opt/ring /opt/ring.c
    rm /opt/ring.c

%environment
    export PATH=/opt:$PATH

You can download both files here: ring.c, mpi_ring.def.

Build the container (both files must be in the same directory):

apptainer build mpi_ring.sif mpi_ring.def

Running with SLURM¶

The MPI library inside the container (Ubuntu's OpenMPI) is not built with SLURM/PMIx support, so srun cannot spawn MPI ranks directly. Instead, you let SLURM allocate the resources and launch a single task, and then mpirun inside the container spawns the MPI processes:

--ntasks=1: SLURM launches one task (the container).
--cpus-per-task=4: allocates 4 CPU cores for that task.
mpirun --oversubscribe -np 4: the container's mpirun spawns 4 ranks. The --oversubscribe flag is needed because OpenMPI sees SLURM_NTASKS=1 and refuses to launch more processes than slots.

The MPI ranks communicate through shared memory within the node.

Why does the container's OpenMPI care about SLURM variables?

OpenMPI has built-in code to detect resource managers by reading environment variables (SLURM_*, PBS_*, LSF_*, etc.). This is passive auto-detection that does not require any integration; the variables are inherited by every child process (srun → apptainer exec → mpirun). OpenMPI sees SLURM_NTASKS=1 and concludes it only has 1 slot.

mpi_ring.slurm

#!/bin/bash
#SBATCH --qos=test
#SBATCH --job-name=mpi-ring
#SBATCH --nodes=1
#SBATCH --ntasks=1
#SBATCH --cpus-per-task=4
#SBATCH --mem=4gb
#SBATCH --time=00:05:00
#SBATCH --output=%x.%j.out
#SBATCH --error=%x.%j.err

module load Apptainer

srun apptainer exec /scratch/$USER/containers/mpi_ring.sif \
    mpirun --oversubscribe -np 4 ring

Expected output:

Rank 0 starts token = 42 (sending to 4 ranks)
Rank 1 received token, added 1, sending token = 43
Rank 2 received token, added 2, sending token = 45
Rank 3 received token, added 3, sending token = 48
Rank 0 received token = 48 (after full ring)

Multi-node MPI

Running MPI across multiple nodes from a container is possible but significantly more complex. It requires:

An MPI implementation inside the container that is ABI-compatible with the host's MPI (same major version).
Bind-mounting the host's MPI libraries, InfiniBand drivers (libibverbs), SLURM's PMI libraries (libpmi, libmunge), and their dependencies into the container at runtime.
Carefully managing LD_LIBRARY_PATH so the container uses the host's MPI transport layer (UCX, InfiniBand) while keeping its own system libraries (libc, etc.) to avoid version conflicts.

This setup is inherently fragile: the exact set of libraries and paths depends on the cluster's software stack, and it can break silently when MPI, SLURM, or the kernel drivers are upgraded. Contact support-hpc@dipc.org if you need multi-node MPI with containers.

Reference: multi-node MPI on Hyperion (OpenMPI 4.1.6)

This example uses the same mpi_ring.sif container from above (built with Ubuntu's OpenMPI 4.1.x), but replaces the container's MPI transport with the host's libraries at runtime. Tested on Hyperion as of March 2026.

The host's OpenMPI, UCX, and system libraries are bind-mounted into the container, and LD_LIBRARY_PATH is set so the host's MPI libraries take priority over the container's:

module load OpenMPI/4.1.6-GCC-13.2.0

srun -N2 -n8 --ntasks-per-node=4 -q test \
    apptainer exec \
    --bind /scicomp/builds:/scicomp/builds \
    --bind /usr/lib64/libmunge.so.2:/usr/lib64/libmunge.so.2 \
    --bind /usr/lib64/libpmi.so.0:/usr/lib64/libpmi.so.0 \
    --bind /usr/lib64/libpmi2.so.0:/usr/lib64/libpmi2.so.0 \
    --bind /usr/lib64/libibverbs.so.1:/usr/lib64/libibverbs.so.1 \
    --bind /usr/lib64/slurm/libslurm_pmi.so:/usr/lib64/slurm/libslurm_pmi.so \
    mpi_ring.sif \
    bash -c "export LD_LIBRARY_PATH=$EBROOTOPENMPI/lib:$EBROOTUCX/lib:$EBROOTLIBEVENT/lib:$EBROOTHWLOC/lib:/usr/lib64:\$LD_LIBRARY_PATH && /opt/ring"

Rank 0 starts token = 42 (sending to 8 ranks)
Rank 1 received token, added 1, sending token = 43
Rank 2 received token, added 2, sending token = 45
Rank 3 received token, added 3, sending token = 48
Rank 4 received token, added 4, sending token = 52
Rank 5 received token, added 5, sending token = 57
Rank 6 received token, added 6, sending token = 63
Rank 7 received token, added 7, sending token = 70
Rank 0 received token = 70 (after full ring)

Key points:

The container's OpenMPI (4.1.x from Ubuntu) is ABI-compatible with the host's OpenMPI 4.1.6.
The host libraries ($EBROOTOPENMPI/lib, etc.) go first in LD_LIBRARY_PATH to override the container's MPI.
System libraries (munge, pmi, ibverbs) are mounted individually at /usr/lib64/ to avoid pulling in the host's libc, which would cause glibc version conflicts.
OpenMPI 4.1.x uses PMI2 for process management (not PMIx). This will change in OpenMPI 5.x.