SLURM workload manager¶

SLURM is the job scheduler and resource manager on all DIPC clusters. You write a batch script describing what resources you need and what to run, submit it with sbatch, and SLURM queues it until resources are available.

Batch scripts¶

A batch script is a shell script with #SBATCH directives that tell SLURM what resources to allocate. Here is a minimal example:

#!/bin/bash
#SBATCH --job-name=my_job
#SBATCH --partition=general
#SBATCH --qos=regular
#SBATCH --nodes=1
#SBATCH --ntasks=1
#SBATCH --cpus-per-task=1
#SBATCH --mem=4gb
#SBATCH --time=01:00:00
#SBATCH --output=%x.%j.out
#SBATCH --error=%x.%j.err

module load some_software/1.0

srun ./my_program < input.dat

Each #SBATCH line is a resource directive. SLURM reads them before executing the script body.

Common directives¶

Submitting and running jobs¶

sbatch - batch submission¶

Submit a script to the queue:

sbatch my_script.slurm
Submitted batch job 123456

The job runs asynchronously. Output goes to the files specified with --output and --error (or slurm-<jobid>.out by default).

salloc interactive allocation¶

Request resources without a batch script. SLURM grants an allocation and drops you into a subshell:

$ salloc --qos=test --nodes=1 --mem=20gb --time=00:10:00
salloc: Granted job allocation 1999

$ srun hostname
hyperion-125

$ exit
salloc: Relinquishing job allocation 1999

To get a shell directly on the compute node, append srun --pty bash:

$ salloc --qos=test --time=00:10:00 srun --pty bash
salloc: Granted job allocation 1477543
salloc: Nodes hyperion-125 are ready for job
user@hyperion-125:~$

srun - direct execution¶

Run a command on allocated resources. Useful for quick tests:

$ srun --qos=test --mem=1gb --time=00:05:00 bash -c 'echo "Hello from $(hostname)"'
Hello from atlas-281

srun blocks until the command finishes. If your terminal disconnects, the job ends. For anything longer than a quick test, wrap srun in a batch script and use sbatch.

Within a batch script, srun is used to launch your program on the allocated resources. It automatically inherits the resource allocation from the #SBATCH directives.

Partitions and QoS¶

Each cluster organizes nodes into partitions and associates them with Quality of Service levels (QoS) that set limits on walltime, node count, and running jobs.

The preemption partition¶

All clusters have a preemption partition containing nodes owned by specific research groups. These nodes are often idle and are made available to all users, with one condition: if the owner group submits a job that needs its nodes, preemption jobs running on those nodes will be canceled and requeued.

The submission procedure is:

#SBATCH --partition=preemption
#SBATCH --qos=regular       # use any QoS from the general partition

Job templates¶

These templates work on all clusters unless noted otherwise. Adjust --ntasks-per-node and --cpus-per-task to match the core count of your target nodes (see the specifications page of each cluster).

MPI¶

#!/bin/bash
#SBATCH --qos=regular
#SBATCH --job-name=mpi_job
#SBATCH --nodes=2
#SBATCH --ntasks-per-node=48
#SBATCH --cpus-per-task=1
#SBATCH --mem=200gb
#SBATCH --time=1-00:00:00
#SBATCH --output=%x.%j.out
#SBATCH --error=%x.%j.err

module load program/version

srun ./binary < input

This runs 96 MPI processes (2 nodes x 48 tasks/node). srun handles process placement and inherits the SLURM allocation automatically.

mpirun -np ${SLURM_NTASKS} ./binary < input

For more details on MPI compilation and execution, see MPI.

OpenMP¶

#!/bin/bash
#SBATCH --qos=regular
#SBATCH --job-name=omp_job
#SBATCH --nodes=1
#SBATCH --ntasks=1
#SBATCH --cpus-per-task=48
#SBATCH --mem=200gb
#SBATCH --time=1-00:00:00
#SBATCH --output=%x.%j.out
#SBATCH --error=%x.%j.err

module load program/version

export OMP_NUM_THREADS=${SLURM_CPUS_PER_TASK}

srun ./binary < input

Always set OMP_NUM_THREADS from SLURM_CPUS_PER_TASK to ensure the thread count matches the allocation. See OpenMP for thread affinity and tuning.

Hybrid MPI + OpenMP¶

#!/bin/bash
#SBATCH --qos=regular
#SBATCH --job-name=hybrid_job
#SBATCH --nodes=2
#SBATCH --ntasks-per-node=4
#SBATCH --cpus-per-task=12
#SBATCH --mem=200gb
#SBATCH --time=1-00:00:00
#SBATCH --output=%x.%j.out
#SBATCH --error=%x.%j.err

module load program/version

export OMP_NUM_THREADS=${SLURM_CPUS_PER_TASK}

srun ./binary < input

This runs 8 MPI processes (4 per node), each with 12 OpenMP threads, using 48 cores per node total. On Atlas FDR, replace srun with mpirun -np ${SLURM_NTASKS} (see MPI note above). See MPI: Hybrid MPI+OpenMP for details.

GPU¶

GPU request syntax differs between clusters. Atlas FDR has no GPUs. For a complete guide including available hardware, constraints, and multi-GPU training, see GPU accelerators.

#!/bin/bash
#SBATCH --qos=regular
#SBATCH --job-name=gpu_job
#SBATCH --nodes=1
#SBATCH --ntasks=1
#SBATCH --cpus-per-task=8
#SBATCH --gres=gpu:1
#SBATCH --constraint=rtx3090
#SBATCH --mem=32gb
#SBATCH --time=04:00:00
#SBATCH --output=%x.%j.out
#SBATCH --error=%x.%j.err

module load CUDA/12.4.0

srun ./my_gpu_program

#!/bin/bash
#SBATCH --qos=regular
#SBATCH --job-name=gpu_job
#SBATCH --nodes=1
#SBATCH --ntasks=1
#SBATCH --cpus-per-task=4
#SBATCH --gres=gpu:p40:1
#SBATCH --mem=32gb
#SBATCH --time=04:00:00
#SBATCH --output=%x.%j.out
#SBATCH --error=%x.%j.err

module load CUDA/11.7.0

srun ./my_gpu_program

Node selection¶

Hyperion: microarchitecture constraints¶

Hyperion has three CPU microarchitectures. By default, SLURM ensures all nodes in a job share the same architecture (no mixing), but you can request a specific one:

#SBATCH --constraint=icelake

#SBATCH --constraint=cascadelake

#SBATCH --constraint=emerald

This constraint works with sbatch, srun, and salloc. Requesting a GPU type via --constraint implicitly selects the CPU architecture of those nodes. See GPU accelerators: GPU and CPU microarchitecture for the mapping.

Atlas EDR and Atlas FDR¶

These clusters have homogeneous CPU architectures within each cluster, so no microarchitecture constraint is needed. See the Atlas EDR specifications and Atlas FDR specifications for node details.

Job arrays¶

Job arrays submit multiple instances of the same job, each with a unique index. They are ideal when you need to run the same computation on many different inputs. See the Job arrays page for a full guide with practical examples.

Dependency chains¶

Start a job only after another job reaches a certain state:

$ sbatch first_job.slurm
Submitted batch job 1000

$ sbatch --dependency=afterok:1000 second_job.slurm
Submitted batch job 1001

Available dependency types:

The singleton dependency is useful for chaining restartable jobs: submit the same script multiple times and each instance waits for the previous one to finish.

Managing jobs¶

squeue - view the queue¶

# All your jobs
squeue -u $USER

# Only running jobs
squeue -u $USER -t RUNNING

# Only pending jobs
squeue -u $USER -t PENDING

# Jobs in a specific partition
squeue -u $USER -p preemption

scancel - cancel jobs¶

# Cancel a specific job
scancel 123456

# Cancel all your jobs
scancel -u $USER

# Cancel all your pending jobs
scancel -t PENDING -u $USER

sacct - job accounting (completed jobs)¶

After a job finishes, sacct shows what actually happened:

sacct -j 123456 --format=JobID,JobName,MaxRSS,Elapsed,State,NodeList

sstat - live job statistics¶

For a currently running job:

sstat --format=JobID,AveCPU,AveRSS,MaxRSS -j 123456

seff - efficiency report¶

Quick summary of CPU and memory efficiency for a completed job:

$ seff 123456
Job ID: 123456
State: COMPLETED
Cores: 48
CPU Utilized: 12:34:56
CPU Efficiency: 87.5%
Memory Utilized: 24.5 GB
Memory Efficiency: 51.0% of 48.00 GB

Local scratch for I/O-intensive jobs¶

Every compute node has a local disk mounted at /lscratch. It is faster than the shared /scratch filesystem because I/O does not travel over the network. Use it when your job reads or writes many small files (e.g., molecular dynamics restarts, image datasets, temporary databases).

The pattern is simple: copy your input to /lscratch at the start of the job, run there, and copy the results back to /scratch at the end.

#!/bin/bash
#SBATCH --qos=regular
#SBATCH --job-name=io_job
#SBATCH --nodes=1
#SBATCH --ntasks=1
#SBATCH --cpus-per-task=8
#SBATCH --mem=16gb
#SBATCH --time=06:00:00
#SBATCH --output=%x.%j.out
#SBATCH --error=%x.%j.err

WORKDIR=/lscratch/${USER}/${SLURM_JOB_ID}
mkdir -p ${WORKDIR}

# Copy input data to local disk
cp /scratch/${USER}/project/input.dat ${WORKDIR}/

# Run from local disk
cd ${WORKDIR}
srun ./my_program < input.dat

# Copy results back to shared storage
cp ${WORKDIR}/output.dat /scratch/${USER}/project/

# Clean up
rm -rf ${WORKDIR}

Requeuing policy¶

If a node fails (freeze, reboot, memory error), SLURM will attempt to requeue your job by default. This is useful for checkpoint-based workflows but dangerous otherwise, because the restarted job will overwrite output files.

To prevent automatic requeuing:

#SBATCH --no-requeue

SLURM environment variables¶

These variables are available inside your batch script at runtime:

Use these to make your scripts portable. For example, export OMP_NUM_THREADS=${SLURM_CPUS_PER_TASK} ensures your thread count always matches the allocation regardless of what you requested.

Job states¶

Troubleshooting¶

squeue -j <jobid> -o "%.18i %.30j %.10R"

scontrol show node <nodename> | grep State
sinfo -N -o "%N %T"

See also¶