Apptainer¶

Apptainer is the container runtime available on our clusters. It allows you to package your software, dependencies, and environment into a single portable image that runs at native performance; with no virtualization overhead. This is especially useful when you need a specific software stack that differs from what is installed on the cluster, or when you want to ensure your results are reproducible across different systems.

Apptainer integrates natively with SLURM and NVIDIA GPUs, making it well suited for HPC workloads.

Loading Apptainer¶

module load Apptainer

Cache and storage¶

Apptainer caches downloaded layers and temporary build files in ~/.apptainer/cache by default. This can quickly fill your home directory quota. Redirect the cache to /scratch before pulling or building anything:

export APPTAINER_CACHEDIR=/scratch/$USER/.apptainer-cache
mkdir -p $APPTAINER_CACHEDIR

To see how much space the cache is using:

apptainer cache list

To clean it:

apptainer cache clean

Store your .sif images on /scratch as well. A typical layout:

/scratch/$USER/
├── containers/          # SIF images
│   ├── pytorch.sif
│   └── my_app.sif
└── .apptainer-cache/    # layer cache (can be cleaned)

Pulling images¶

You can pull images directly from Docker Hub. Apptainer converts them to its own SIF format:

apptainer pull docker://ubuntu:latest

This creates a file called ubuntu_latest.sif in your current directory.

Apptainer supports any OCI-compatible registry, not just Docker Hub. For example, NVIDIA NGC provides GPU-optimized containers for deep learning, HPC, and scientific computing:

apptainer pull docker://nvcr.io/nvidia/pytorch:25.04-py3
apptainer pull docker://nvcr.io/nvidia/tensorflow:25.04-tf2-py3
apptainer pull docker://nvcr.io/hpc/gromacs:2024.3

Other registries work the same way:

# GitHub Container Registry
apptainer pull docker://ghcr.io/org/image:tag

# Quay.io
apptainer pull docker://quay.io/org/image:tag

Choosing a base image¶

Operating system¶

Ubuntu is the most common base image for HPC containers. It does not need to match the host OS; the container has its own userspace and only shares the kernel with the host. Ubuntu has the broadest package availability and most scientific software installation guides are written for it.

Bootstrap: docker
From: ubuntu:24.04

For GPU workloads, start from NVIDIA NGC images instead of a bare OS. They come with CUDA, cuDNN, and NCCL pre-configured and optimized:

Bootstrap: docker
From: nvcr.io/nvidia/pytorch:25.04-py3

Conda environments inside containers¶

Conda environments generate tens of thousands of small files. On parallel filesystems this creates significant overhead: slow environment activation, heavy metadata load, and rapid inode quota consumption.

Packaging a conda environment inside a container solves this. The entire environment becomes a single .sif file, which the filesystem handles efficiently. To build one:

Bootstrap: docker
From: condaforge/miniforge3:latest

%files
    environment.yml /opt/environment.yml

%post
    mamba env create -f /opt/environment.yml -p /opt/conda-env
    mamba clean -afy
    rm /opt/environment.yml

%environment
    export PATH=/opt/conda-env/bin:$PATH

apptainer build conda_env.sif conda_env.def
apptainer exec conda_env.sif python my_script.py

Summary¶

Running commands in a container¶

Use apptainer exec to run a command inside a container:

apptainer exec ubuntu_latest.sif cat /etc/os-release

apptainer exec docker://ubuntu:latest cat /etc/os-release

To get an interactive shell inside the container:

apptainer shell ubuntu_latest.sif

Type exit to return to the host.

Bind mounts¶

By default, Apptainer mounts your home directory ($HOME), /tmp, and the current working directory inside the container. If your data lives elsewhere (which it should on an HPC cluster), you need to bind those paths explicitly with --bind:

apptainer exec --bind /scratch/$USER:/scratch/$USER my_container.sif python analyze.py

This makes /scratch/$USER available at the same path inside the container. You can bind multiple directories:

apptainer exec \
    --bind /scratch/$USER:/scratch/$USER \
    --bind /data/shared:/data/shared \
    my_container.sif python analyze.py

You can also map a host path to a different path inside the container:

apptainer exec --bind /scratch/$USER/input:/mnt/input my_container.sif cat /mnt/input/data.csv

apptainer exec --bind /scratch/$USER:/scratch/$USER my_container.sif ls /scratch/$USER/

Environment variables¶

Apptainer passes most host environment variables into the container by default. To set or override a variable specifically for the container, prefix it with APPTAINERENV_:

export APPTAINERENV_OMP_NUM_THREADS=4
apptainer exec my_container.sif ./my_program

Inside the container, OMP_NUM_THREADS will be set to 4. This works with any variable:

# Control thread count for OpenMP applications
export APPTAINERENV_OMP_NUM_THREADS=8

# Set CUDA visible devices
export APPTAINERENV_CUDA_VISIBLE_DEVICES=0,1

# Pass custom configuration to your application
export APPTAINERENV_MY_APP_CONFIG=/data/config.yaml

You can also pass them inline:

APPTAINERENV_OMP_NUM_THREADS=4 apptainer exec my_container.sif ./my_program

Variables defined in the %environment section of the definition file are set at runtime inside the container. APPTAINERENV_* variables from the host take precedence over %environment values.

Building custom containers¶

Definition files (.def) are recipes that specify how to build a container. A minimal example:

Bootstrap: docker
From: ubuntu:24.04

%post
    apt-get update && apt-get install -y vim

%environment
    export MYVAR=myvalue

Build it with:

apptainer build my_container.sif my_container.def

The main sections of a definition file are:

GPU support¶

To give the container access to the host's NVIDIA GPUs, add the --nv flag:

apptainer exec --nv my_container.sif python my_script.py

Without --nv, the container will not see any GPUs:

# No --nv flag → no GPUs visible
apptainer exec pytorch.sif python torch-test.py
PyTorch Version: 2.5.1+cu124
CUDA available: False
Running on CPU.
Epoch  20/100 - Loss: 0.0026
Epoch  40/100 - Loss: 0.0019
Epoch  60/100 - Loss: 0.0016
Epoch  80/100 - Loss: 0.0014
Epoch 100/100 - Loss: 0.0013

Prediction for input 10: 19.85 (expected: ~20.00)

# With --nv → GPUs detected
apptainer exec --nv pytorch.sif python torch-test.py
PyTorch Version: 2.5.1+cu124
CUDA available: True
GPU: NVIDIA GeForce RTX 3090
Epoch  20/100 - Loss: 0.0722
Epoch  40/100 - Loss: 0.0622
Epoch  60/100 - Loss: 0.0543
Epoch  80/100 - Loss: 0.0474
Epoch 100/100 - Loss: 0.0414

Prediction for input 10: 19.16 (expected: ~20.00)

Example: PyTorch with GPU¶

Definition file (pytorch.def):

Bootstrap: docker
From: pytorch/pytorch:2.6.0-cuda12.6-cudnn9-runtime

%post
    pip install --upgrade pip

Build and run:

apptainer build pytorch.sif pytorch.def
apptainer exec --nv pytorch.sif python torch-test.py

You can download the test script used in this example here: torch-test.py.

SLURM integration¶

You can run Apptainer containers as SLURM jobs like any other command. GPU requests differ between clusters:

#!/bin/bash
#SBATCH --qos=test
#SBATCH --job-name=torch-test
#SBATCH --gres=gpu:1
#SBATCH --constraint=rtx3090
#SBATCH --mem=10gb
#SBATCH --output=%x-%j.out
#SBATCH --error=%x-%j.err

module load Apptainer

srun apptainer exec --nv pytorch.sif python torch-test.py

#!/bin/bash
#SBATCH --qos=test
#SBATCH --job-name=torch-test
#SBATCH --gres=gpu:rtx3090:1
#SBATCH --mem=10gb
#SBATCH --output=%x-%j.out
#SBATCH --error=%x-%j.err

module load Apptainer

srun apptainer exec --nv pytorch.sif python torch-test.py

Submit it with:

sbatch my_job.sh

Next steps¶

For writable overlays, MPI inside containers, and multi-node MPI, see Advanced topics.