Nextflow¶

Nextflow is a workflow manager that submits each step of your pipeline as a separate SLURM job. You define the steps and their data dependencies; Nextflow handles job submission, parallelism, staging files, and retrying failed tasks. If a pipeline fails halfway, you can resume from where it left off without re-running completed steps.

For a general introduction to the Nextflow language (processes, channels, operators), see the official documentation. This page focuses on running Nextflow on our clusters.

Loading Nextflow¶

module load Nextflow/25.04.6

Java is loaded automatically as a dependency. Quick test to verify the installation:

process GREET {
    input:
    val name

    output:
    stdout

    script:
    """
    echo "Hello, ${name}!"
    """
}

workflow {
    names = Channel.of("Alice", "Bob", "Charlie")
    GREET(names) | view
}

nextflow run hello.nf

 N E X T F L O W   ~  version 25.04.6

Launching `hello.nf` [hopeful_panini] DSL2 - revision: 9a7ae29534

executor >  local (3)
[6c/f5f811] GREET (3) [100%] 3 of 3 ✔
Hello, Alice!

Hello, Bob!

Hello, Charlie!

SLURM configuration¶

Nextflow needs a nextflow.config file in your project directory to know how to submit jobs. At minimum:

process {
    executor = "slurm"
    queue    = "general"
    cpus     = 4
    memory   = "8 GB"
    time     = "1d"
}

With this configuration, every task in your pipeline becomes a SLURM job submitted via sbatch. Nextflow monitors their state and launches downstream tasks as dependencies are satisfied.

process {
    queue          = "general"
    clusterOptions = "--qos=long"
}

Work directory¶

By default, Nextflow creates a work/ directory in the current folder. On the cluster, point it to /scratch for better performance and to avoid filling your home quota:

nextflow run pipeline.nf -w /scratch/<username>/nf-work

Or set it permanently in nextflow.config:

workDir = "/scratch/<username>/nf-work"

Each task gets a subdirectory inside work/ containing the script (.command.sh), logs (.command.log), and output files. These are essential for debugging.

Per-process resources¶

Different steps usually need different resources. Use labels in your pipeline and configure them in nextflow.config:

process {
    executor = "slurm"
    queue    = "general"

    withLabel: small {
        cpus   = 2
        memory = "4 GB"
        time   = "1h"
    }

    withLabel: big {
        cpus   = 16
        memory = "64 GB"
        time   = "12h"
    }
}

You can also target a specific process by name:

process {
    withName: ALIGN {
        cpus           = 16
        memory         = "64 GB"
        time           = "2d"
        clusterOptions = "--qos=long"
    }
}

Dynamic resources with automatic retry¶

Tasks that run out of memory are killed by SLURM (exit code 137). A common pattern is to increase resources on each retry:

process {
    executor      = "slurm"
    errorStrategy = { task.exitStatus == 137 ? "retry" : "terminate" }
    maxRetries    = 3

    memory = { 8.GB * task.attempt }
    time   = { 2.h  * task.attempt }
}

First attempt: 8 GB / 2 h. Second: 16 GB / 4 h. Third: 24 GB / 6 h.

GPU jobs¶

Use clusterOptions to request GPUs. On Hyperion, use --constraint to select a specific GPU type:

process {
    withName: INFERENCE {
        queue          = "general"
        clusterOptions = "--gres=gpu:1 --constraint=a100-pcie"
        cpus           = 8
        memory         = "32 GB"
        time           = "1d"
    }
}

process {
    withName: SMALL_GPU_TASK {
        queue          = "preemption-gpu"
        clusterOptions = "--gres=gpu:1 --qos=preemption-gpu"
        cpus           = 4
        memory         = "32 GB"
        time           = "1d"
    }
}

process {
    withName: INFERENCE {
        queue          = "general"
        clusterOptions = "--gres=gpu:rtx3090:1"
        cpus           = 8
        memory         = "32 GB"
        time           = "1d"
    }
}

Profiles¶

Profiles let you switch between clusters without changing the pipeline:

profiles {
    hyperion {
        process.executor = "slurm"
        process.queue    = "general"
        workDir          = "/scratch/<username>/nf-work"
    }

    atlas {
        process.executor = "slurm"
        process.queue    = "general"
        workDir          = "/scratch/<username>/nf-work"
    }

    local {
        process.executor = "local"
        process.cpus     = 4
    }
}

Select a profile at runtime:

nextflow run pipeline.nf -profile hyperion

Using containers¶

Nextflow integrates with Apptainer natively. When you specify a container, each task runs inside it automatically.

Configuration¶

Apptainer must be loaded both in your session (Nextflow pulls images from the login node) and inside each SLURM task:

module load Apptainer

apptainer.enabled    = true
apptainer.autoMounts = true
apptainer.runOptions = "--nv"
apptainer.cacheDir   = "/scratch/<username>/.apptainer-cache"

process {
    beforeScript = "module load Apptainer"
}

The --nv flag exposes the host GPU drivers inside the container, which is required for any GPU workload. It is harmless on CPU-only tasks.

Specifying containers¶

Per process in nextflow.config:

process {
    withName: ALIGN {
        container = "docker://biocontainers/bwa:v0.7.17"
    }
    withName: QUANTIFY {
        container = "/scratch/<username>/containers/salmon.sif"
    }
}

When you use a docker:// URI, Nextflow pulls the image and converts it to SIF format automatically. Pre-built .sif files can be referenced directly by path.

Complete example: SLURM + Apptainer on Hyperion¶

A minimal pipeline that runs one CPU task and one GPU task, each inside its own container:

process CPU_TASK {
    label "cpu"

    output:
    stdout

    script:
    """
    echo "Running on \$(hostname) with \$(nproc) CPUs"
    cat /etc/os-release | grep PRETTY_NAME
    """
}

process GPU_TASK {
    label "gpu"

    output:
    stdout

    script:
    """
    echo "Running on \$(hostname)"
    nvidia-smi --query-gpu=name,memory.total --format=csv,noheader
    """
}

workflow {
    CPU_TASK() | view
    GPU_TASK() | view
}

apptainer.enabled    = true
apptainer.autoMounts = true
apptainer.runOptions = "--nv"
apptainer.cacheDir   = "/scratch/<username>/.apptainer-cache"
workDir              = "/scratch/<username>/nf-work"

process {
    executor     = "slurm"
    queue        = "general"
    beforeScript = "module load Apptainer"

    withLabel: cpu {
        cpus      = 4
        memory    = "8 GB"
        time      = "2h"
        container = "docker://ubuntu:24.04"
    }

    withLabel: gpu {
        cpus           = 8
        memory         = "32 GB"
        time           = "6h"
        clusterOptions = "--gres=gpu:1 --constraint=a100-pcie"
        container      = "docker://nvidia/cuda:12.4.1-base-ubuntu22.04"
    }
}

Place both files in the same directory and run the pipeline:

module load Nextflow/25.04.6 Apptainer
nextflow run demo.nf

Nextflow submits two SLURM jobs (one per task), pulls the container images on first run, and prints the results as each task completes:

 N E X T F L O W   ~  version 25.04.6

Launching `demo.nf` [admiring_stone] DSL2 - revision: 8ae07c8af1

executor >  slurm (2)
[62/8337c2] CPU_TASK [100%] 1 of 1 ✔
[59/98990a] GPU_TASK [100%] 1 of 1 ✔
Running on hyperion-257
NVIDIA A100 80GB PCIe, 81920 MiB

Completed at: 16-Mar-2026 14:41:52
Duration    : 48m 55s
CPU hours   : (a few seconds)
Succeeded   : 2

The GPU task ran on hyperion-257 and correctly detected the A100 PCIe GPU inside the container.

Running nf-core pipelines¶

nf-core provides production-ready pipelines for common bioinformatics analyses. These pipelines include both singularity and apptainer profiles that handle container downloads automatically.

Set the cache directory before running:

export NXF_APPTAINER_CACHEDIR=/scratch/<username>/.apptainer-cache
mkdir -p $NXF_APPTAINER_CACHEDIR

Example with nf-core/rnaseq (see the nf-core/rnaseq usage docs for the samplesheet format and available options):

module load Nextflow/25.04.6 Apptainer

nextflow run nf-core/rnaseq \
    -r 3.14.0 \
    -profile apptainer \
    --input samplesheet.csv \
    --outdir results \
    --genome GRCh38 \
    -w /scratch/<username>/nf-work

Some popular nf-core pipelines:

Resuming failed pipelines¶

If a pipeline fails, fix the issue and re-run with -resume:

nextflow run pipeline.nf -resume

Nextflow reuses cached results from work/ and only re-executes failed or new tasks.

Monitoring¶

Generate an HTML report with resource usage per task:

nextflow run pipeline.nf -with-report report.html -with-timeline timeline.html

View past runs:

nextflow log

Cleaning up¶

The work/ directory grows quickly. After a successful run, clean it:

nextflow clean -f

Or remove only runs older than a specific one:

nextflow clean -f -before <run_name>

Troubleshooting¶

module load Nextflow/25.04.6

sacctmgr show qos format=name,maxwall,maxtresperuser