Job arrays¶

Job arrays let you submit many instances of the same job with a single sbatch command. Each instance gets a unique index in the $SLURM_ARRAY_TASK_ID variable, which you use to select the corresponding input file, parameter, or dataset. SLURM manages them as a single entity, so they are easier to monitor and cancel than hundreds of individual jobs.

Use job arrays when you need to:

Run the same program on many input files
Sweep a parameter space (one value per task)
Process independent datasets in parallel

Basic syntax¶

array_job.slurm

#!/bin/bash
#SBATCH --qos=regular
#SBATCH --job-name=array_job
#SBATCH --nodes=1
#SBATCH --ntasks=1
#SBATCH --cpus-per-task=1
#SBATCH --mem=4gb
#SBATCH --time=01:00:00
#SBATCH --array=0-9
#SBATCH --output=%x-%A_%a.out
#SBATCH --error=%x-%A_%a.err

echo "Array task ID: ${SLURM_ARRAY_TASK_ID}"
./process_input input_${SLURM_ARRAY_TASK_ID}.dat

sbatch array_job.slurm

This submits 10 independent tasks (indexes 0 through 9). Each one runs the same script but with a different value of $SLURM_ARRAY_TASK_ID.

Info

job-arrays are limited to 10.000 tasks per-job and indexes must be lower than 4.000.000

Output filenames¶

Use %A (array job ID) and %a (task index) in --output and --error to get separate files per task:

Pattern	Expands to
`%A`	Array parent job ID (same for all tasks)
`%a`	Task index (unique per task)
`%x`	Job name

Example: --output=%x-%A_%a.out produces array_job-123456_0.out, array_job-123456_1.out, etc.

Index patterns¶

#SBATCH --array=1-100        # indexes 1 through 100
#SBATCH --array=1-100%10     # same, but max 10 running simultaneously
#SBATCH --array=0,3,7,15     # specific indexes
#SBATCH --array=1-20,50-60   # ranges can be combined

The %N throttle is useful to avoid flooding the scheduler or a shared filesystem with too many concurrent jobs.

Common use cases¶

One input file per task¶

The simplest case: your input files are numbered and each task processes one.

#SBATCH --array=1-50

./simulate input_${SLURM_ARRAY_TASK_ID}.dat

This requires files named input_1.dat, input_2.dat, ..., input_50.dat.

Reading parameters from a file¶

When your inputs are not numbered or you want to sweep parameters, list them in a text file and use sed to pick the right line:

params.txt

0.01  100  alpha
0.05  200  beta
0.10  500  gamma
0.50  1000 delta

param_sweep.slurm

#!/bin/bash
#SBATCH --qos=regular
#SBATCH --job-name=sweep
#SBATCH --array=1-4
#SBATCH --nodes=1
#SBATCH --ntasks=1
#SBATCH --cpus-per-task=1
#SBATCH --mem=4gb
#SBATCH --time=02:00:00
#SBATCH --output=%x-%A_%a.out
#SBATCH --error=%x-%A_%a.err

# Read the line corresponding to this task (line 1 for task 1, etc.)
PARAMS=$(sed -n "${SLURM_ARRAY_TASK_ID}p" params.txt)
RATE=$(echo $PARAMS | awk '{print $1}')
STEPS=$(echo $PARAMS | awk '{print $2}')
LABEL=$(echo $PARAMS | awk '{print $3}')

echo "Running with rate=${RATE}, steps=${STEPS}, label=${LABEL}"
./my_program --rate ${RATE} --steps ${STEPS} --output result_${LABEL}.dat

Tip

Start the array at 1 (not 0) when reading from a file, since sed -n "1p" gives the first line.

Processing a list of files¶

When your files have arbitrary names, list them and index into the list:

filelist.txt

/scratch/${USER}/data/sample_A.fasta
/scratch/${USER}/data/sample_B.fasta
/scratch/${USER}/data/sample_C.fasta

process_files.slurm

#!/bin/bash
#SBATCH --qos=regular
#SBATCH --job-name=process
#SBATCH --array=1-3
#SBATCH --nodes=1
#SBATCH --ntasks=1
#SBATCH --cpus-per-task=4
#SBATCH --mem=8gb
#SBATCH --time=04:00:00
#SBATCH --output=%x-%A_%a.out
#SBATCH --error=%x-%A_%a.err

INPUT=$(sed -n "${SLURM_ARRAY_TASK_ID}p" filelist.txt)

echo "Processing: ${INPUT}"
./analyze --threads ${SLURM_CPUS_PER_TASK} ${INPUT}

Managing job arrays¶

Monitoring¶

# All tasks in an array
squeue -j 123456

# Only pending tasks
squeue -j 123456 -t PENDING

Cancelling¶

# Cancel the entire array
scancel 123456

# Cancel a specific task
scancel 123456_5

# Cancel a range of tasks
scancel 123456_[10-20]

Resubmitting failed tasks¶

If some tasks fail, check which ones and resubmit only those:

# Find failed tasks
sacct -j 123456 --format=JobID,State | grep FAILED

123456_3     FAILED
123456_7     FAILED

# Resubmit only the failed indexes
sbatch --array=3,7 array_job.slurm

Combining arrays with dependencies¶

You can chain a post-processing job that runs after all array tasks complete:

$ sbatch array_job.slurm
Submitted batch job 123456

$ sbatch --dependency=afterok:123456 postprocess.slurm
Submitted batch job 123457

The postprocess.slurm job will start only after all tasks in the array finish successfully.

Tips¶

Use arrays instead of loops

If you find yourself writing a bash loop that calls sbatch many times, replace it with a job array. Arrays are lighter on the scheduler, easier to manage, and simpler to resubmit on failure.

Throttle concurrent tasks

Use --array=1-1000%50 to limit to 50 tasks running at once. This prevents overloading shared resources like filesystems or license servers.

Combine with local scratch

If each task does heavy I/O, use /lscratch inside the array job. See Local scratch for I/O-intensive jobs.