GPU computing stack¶

This page covers the software stack and tools for GPU computing on our clusters. For how to request GPUs through SLURM, see GPU accelerators.

CUDA toolkit¶

A base CUDA toolkit is installed system-wide alongside the GPU driver. The module system provides additional CUDA versions and related libraries (cuDNN, cuBLAS, NCCL) that you can load independently:

module load CUDA/12.4.0

Verify the driver and toolkit versions:

nvidia-smi          # driver version, GPU model, memory usage
nvcc --version      # CUDA compiler version

What CUDA versions are available?

module avail CUDA

Always match the CUDA version to your application's requirements. Newer is not always better; some software requires specific CUDA versions.

cuDNN¶

cuDNN (CUDA Deep Neural Network library) provides optimized primitives for deep learning (convolutions, attention, normalization). It is required by PyTorch, TensorFlow, and most deep learning frameworks.

module avail cuDNN
module load cuDNN/8.9.7.29-CUDA-12.4.0

Most users do not need to load cuDNN explicitly; deep learning frameworks installed via conda or containers typically bundle their own cuDNN.

Installing CUDA libraries via conda¶

You can also install CUDA toolkit, cuDNN, and NCCL inside a conda environment:

module load Miniforge3
conda create -n ml python=3.11
conda activate ml
conda install pytorch pytorch-cuda=12.4 -c pytorch -c nvidia

This installs PyTorch with its own CUDA runtime, cuDNN, and NCCL, independent of the system modules. The advantage is a self-contained environment; the downside is that conda environments generate tens of thousands of small files, which can be slow on parallel filesystems and consume inode quota.

Conda environments inside containers

If your conda environment is large or you want faster startup times, consider building it inside an Apptainer container. The entire environment becomes a single .sif file, which eliminates the small-file overhead on /scratch and makes the environment fully portable. See Containers for details.

GPU topology¶

Not all GPU connections are equal. The interconnect between GPUs determines how fast they can exchange data during multi-GPU workloads (distributed training, multi-GPU simulations).

Checking the topology¶

From inside a GPU job, run:

nvidia-smi topo -m

Hyperion: A100 SXM4Hyperion: A100 PCIeAtlas EDR: RTX 3090Atlas EDR: Tesla P40

Full-mesh NVLink. Every GPU pair is connected via NV12 (12 NVLink links). Any GPU can communicate with any other at full NVLink speed.

        GPU0    GPU1    GPU2    GPU3    GPU4    GPU5    GPU6    GPU7
GPU0     X      NV12    NV12    NV12    NV12    NV12    NV12    NV12
GPU1    NV12     X      NV12    NV12    NV12    NV12    NV12    NV12
GPU2    NV12    NV12     X      NV12    NV12    NV12    NV12    NV12
GPU3    NV12    NV12    NV12     X      NV12    NV12    NV12    NV12
GPU4    NV12    NV12    NV12    NV12     X      NV12    NV12    NV12
GPU5    NV12    NV12    NV12    NV12    NV12     X      NV12    NV12
GPU6    NV12    NV12    NV12    NV12    NV12    NV12     X      NV12
GPU7    NV12    NV12    NV12    NV12    NV12    NV12    NV12     X

Paired NVLink + PCIe. Only GPU pairs (0-1, 2-3, 4-5, 6-7) are connected via NVLink. Communication between pairs goes through PCIe Gen4 bridges (PXB within the same socket, SYS across sockets).

        GPU0    GPU1    GPU2    GPU3    GPU4    GPU5    GPU6    GPU7
GPU0     X      NV12    PXB     PXB     SYS     SYS     SYS     SYS
GPU1    NV12     X      PXB     PXB     SYS     SYS     SYS     SYS
GPU2    PXB     PXB      X      NV12    SYS     SYS     SYS     SYS
GPU3    PXB     PXB     NV12     X      SYS     SYS     SYS     SYS
GPU4    SYS     SYS     SYS     SYS      X      NV12    PXB     PXB
GPU5    SYS     SYS     SYS     SYS     NV12     X      PXB     PXB
GPU6    SYS     SYS     SYS     SYS     PXB     PXB      X      NV12
GPU7    SYS     SYS     SYS     SYS     PXB     PXB     NV12     X

2 GPUs per node, same NUMA node, connected via PCIe Gen3 bridges (NODE).

        GPU0    GPU1
GPU0     X      NODE
GPU1    NODE     X

2 GPUs per node, each on a different CPU socket (SYS). Worst case for multi-GPU communication.

        GPU0    GPU1
GPU0     X      SYS
GPU1    SYS     X

Topology legend:

Code	Meaning
NV12	NVLink (12 links); up to 300 GB/s per direction (A100 architecture whitepaper)
PXB	Multiple PCIe bridges (same socket, no CPU traversal)
PIX	Single PCIe bridge
NODE	PCIe + interconnect between PCIe Host Bridges within a NUMA node
PHB	PCIe + CPU (PCIe Host Bridge)
SYS	Cross-socket via QPI/UPI

PCIe generation	Theoretical bandwidth (x16)	Practical bandwidth
Gen3 (Atlas EDR)	16 GB/s	~12 GB/s
Gen4 (Hyperion)	32 GB/s	~25 GB/s

When does topology matter?¶

Workload	Impact
Single GPU	None
Multi-GPU training (DDP, FSDP)	High; NVLink can be 10-20x faster for gradient exchange
Multi-GPU inference	Low to moderate
Single GPU + large model (offloading)	Moderate; host-GPU transfers are PCIe-bound regardless

Tip

If your multi-GPU job runs significantly slower than expected, check the topology first. Running a 2-GPU job on GPUs connected via SYS instead of NV12 can halve your throughput.

NCCL¶

NCCL (NVIDIA Collective Communications Library) is the communication backend used by PyTorch, TensorFlow, and other frameworks for multi-GPU and multi-node GPU operations. It automatically detects the GPU topology and selects the best transport (NVLink, PCIe, InfiniBand).

Key environment variables¶

Variable	Purpose	Common values
`NCCL_DEBUG`	Logging verbosity	`WARN` (default), `INFO` (shows transports chosen), `TRACE` (very verbose)
`NCCL_DEBUG_SUBSYS`	Filter debug output by subsystem	`INIT`, `NET`, `GRAPH`, `ALL`
`NCCL_IB_DISABLE`	Disable InfiniBand transport	`0` (default), `1` (force shared memory/PCIe)
`NCCL_P2P_DISABLE`	Disable peer-to-peer (NVLink/PCIe direct)	`0` (default), `1`
`NCCL_SOCKET_IFNAME`	Network interface for socket-based transport	`ib0`, `eth0`
`NCCL_IB_HCA`	Select specific InfiniBand HCA	`mlx5_0`, `mlx5_1`

Debugging NCCL¶

When multi-GPU training hangs or is unexpectedly slow, start with:

export NCCL_DEBUG=INFO

This prints which transport NCCL selected for each connection. Look for lines like:

NCCL INFO Channel 00 :  0[0] -> 1[1] via P2P/IPC/read
NCCL INFO Channel 00 :  0[0] -> 1[1] via NET/IB/0

P2P/IPC: direct GPU-to-GPU (NVLink or PCIe). Fast.
NET/IB: InfiniBand network. Used for cross-node communication.
NET/Socket: TCP fallback. Very slow; indicates a configuration problem.

NCCL timeout errors

If you see NCCL WARN Cuda failure or timeout errors during distributed training, common causes are:

Firewall or network issues: set NCCL_SOCKET_IFNAME to the correct interface (ib0 for InfiniBand).
Mismatched NCCL versions: ensure all nodes use the same CUDA and NCCL version.
Insufficient shared memory: containers may need --bind /dev/shm:/dev/shm or larger /dev/shm. SLURM allocates shared memory proportional to the requested --mem.

Bandwidth test¶

The nccl-tests suite measures actual GPU-to-GPU bandwidth. To build and run it:

module load CUDA/12.4.0 OpenMPI/4.1.6-GCC-13.2.0

git clone https://github.com/NVIDIA/nccl-tests.git
cd nccl-tests
make MPI=1

Run an all-reduce benchmark on a single node with 8 GPUs:

nccl_bench.slurm

#!/bin/bash
#SBATCH --job-name=nccl-bench
#SBATCH --qos=test
#SBATCH --nodes=1
#SBATCH --ntasks=1
#SBATCH --cpus-per-task=8
#SBATCH --gres=gpu:8
#SBATCH --constraint=a100-sxm4
#SBATCH --mem=64gb
#SBATCH --time=00:10:00
#SBATCH --output=%x.%j.out
#SBATCH --error=%x.%j.err

module load CUDA/12.4.0

./build/all_reduce_perf -b 8 -e 1G -f 2 -g 8

Benchmark results on Hyperion (8 GPUs, message sizes 1-16 GB):

Collective	A100 SXM4 (NVLink)	A100 PCIe
All-reduce	232 GB/s	18 GB/s
All-gather	223 GB/s	17 GB/s
Reduce-scatter	225 GB/s	18 GB/s

The SXM4 nodes achieve ~75% of the theoretical NVLink peak (300 GB/s), while the PCIe nodes are limited by the PCIe Gen4 bridges between GPU pairs. This is a 12x difference in collective bandwidth, which directly impacts multi-GPU training throughput.

Useful environment variables¶

Reference table for common GPU-related environment variables:

Variable	Purpose	Set by
`CUDA_VISIBLE_DEVICES`	GPUs visible to the application	SLURM (automatic)
`NVIDIA_VISIBLE_DEVICES`	GPUs visible inside containers	SLURM / Apptainer `--nv`
`PYTORCH_CUDA_ALLOC_CONF`	PyTorch memory allocator tuning	User
`TF_FORCE_GPU_ALLOW_GROWTH`	TensorFlow: allocate GPU memory on demand	User
`CUDA_LAUNCH_BLOCKING`	Synchronous CUDA execution (for debugging)	User
`TORCH_DISTRIBUTED_DEBUG`	PyTorch distributed debugging	User (`DETAIL`, `INFO`, `OFF`)
`NCCL_DEBUG`	NCCL verbosity	User
`OMP_NUM_THREADS`	CPU threads for hybrid CPU+GPU codes	User

Tip

Set OMP_NUM_THREADS to match --cpus-per-task in your SLURM script. Many GPU applications (PyTorch data loaders, preprocessing) use CPU threads alongside the GPU.

Common issues¶

My GPU job is slower than expected

Check GPU utilization with nvidia-smi or nvidia-smi dmon -s u. Low utilization usually means the GPU is waiting for data (CPU bottleneck, slow I/O, small batch size).
For multi-GPU jobs, check the topology (nvidia-smi topo -m). Cross-socket PCIe connections (SYS) are much slower than NVLink.
Make sure OMP_NUM_THREADS and --cpus-per-task are set appropriately for data loading.

CUDA out of memory (OOM)

Reduce batch size.
For PyTorch, try PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True to reduce memory fragmentation.
Check if other processes are using the GPU: nvidia-smi. SLURM should isolate GPUs, but verify with echo $CUDA_VISIBLE_DEVICES.

nvidia-smi shows no GPUs

Did you request GPUs in your SLURM script? (--gres=gpu:N)
If using a container, did you pass the --nv flag to apptainer exec?
Check echo $CUDA_VISIBLE_DEVICES to see which GPUs SLURM assigned.

NCCL hangs during distributed training

Set NCCL_DEBUG=INFO to see what transport is being used.
For multi-node jobs, ensure NCCL_SOCKET_IFNAME points to the InfiniBand interface (ib0).
Check that all nodes can communicate over InfiniBand: ibstat should show State: Active.
Increase the timeout: NCCL_TIMEOUT=1200 (seconds).