Python¶

Python is the most widely used programming language for scientific computing. On our clusters, Python environments are managed through Miniforge3, a lightweight conda installer that uses the open conda-forge channel by default.

Getting started¶

Load the Miniforge3 module:

module load Miniforge3

This activates a minimal base environment called miniforge3-base and gives you access to python, conda, pip, and mamba (a faster drop-in replacement for conda). Before creating or installing anything, deactivate this base environment:

conda deactivate

Check the Python version:

python --version

Why use conda environments?¶

The base environment provides a minimal Python installation. For your research work, you should create dedicated environments where you can install exactly the packages you need, at the versions you need, without affecting other users or projects.

Environments also make your work reproducible: you can export an environment to a YAML file and recreate it anywhere.

Creating environments¶

Basic environment¶

module load Miniforge3
mamba create --prefix /scratch/$USER/conda-envs/myenv python=3.12

You can include packages at creation time:

mamba create --prefix /scratch/$USER/conda-envs/myenv python=3.12 numpy scipy matplotlib

Activate it by providing the path:

conda activate /scratch/$USER/conda-envs/myenv

envs_dirs:
  - /scratch/<username>/conda-envs

Create an environment from a YAML file¶

You can define an environment in a YAML file and create it in one step:

mamba env create -f environment.yml

name: pytorch-gpu
channels:
  - pytorch
  - nvidia
  - conda-forge
dependencies:
  - python=3.12
  - pytorch=2.5
  - torchvision
  - torchaudio
  - pytorch-cuda=12.4
  - numpy
  - scipy
  - matplotlib
  - pandas
  - jupyterlab

mamba env create -f environment.yml --prefix /scratch/$USER/conda-envs/pytorch-gpu

Activating and deactivating environments¶

Activate an environment:

conda activate myenv

Deactivate it and return to base:

conda deactivate

Installing packages¶

With conda/mamba¶

Search for a package:

mamba search numpy

Install it in the active environment:

mamba install numpy

Install a specific version:

mamba install numpy=1.26

With pip¶

Some packages are not available through conda-forge. In those cases, use pip inside an active conda environment:

conda activate myenv
pip install package-name

Bioinformatics packages (Bioconda)¶

If you work in bioinformatics, genomics, or related fields, the Bioconda channel provides thousands of pre-built packages (samtools, BWA, STAR, DESeq2, snakemake, etc.).

To create an environment with Bioconda packages, specify both the conda-forge and bioconda channels:

mamba create --prefix /scratch/$USER/conda-envs/bioenv \
    -c conda-forge -c bioconda \
    python=3.12 samtools bwa snakemake

Or in an environment.yml:

name: bioenv
channels:
  - conda-forge
  - bioconda
dependencies:
  - python=3.12
  - samtools
  - bwa
  - snakemake

Exporting and reproducing environments¶

Export your current environment to a YAML file:

conda env export > environment.yml

For a more portable export (without build strings tied to your platform):

conda env export --no-builds > environment.yml

Recreate it on another system or after a scratch cleanup:

mamba env create -f environment.yml

Clone an existing environment:

conda create --prefix /scratch/$USER/conda-envs/myclone --clone /scratch/$USER/conda-envs/myenv

Using conda environments in SLURM jobs¶

When running Python in a SLURM job, you need to load the module and activate the environment in your job script:

#!/bin/bash
#SBATCH --job-name=python-job
#SBATCH --qos=regular
#SBATCH --ntasks=1
#SBATCH --mem=4gb
#SBATCH --output=%x-%j.out
#SBATCH --error=%x-%j.err

module load Miniforge3
conda activate /scratch/$USER/conda-envs/myenv

python my_script.py

Two approaches to Python on our clusters¶

There are two ways to use Python on our systems, each with different trade-offs. Understanding the difference will help you choose the right one for your workflow.

Approach 1: Miniforge3 (conda/mamba)¶

This is what most of this page covers. You load Miniforge3, create conda environments, and install packages from conda-forge or pip. This gives you full control over your Python version and packages.

When to use it:

• You need specific package versions or combinations.
• You are working on a project with a requirements.txt or environment.yml.
• You need packages that are not available as pre-installed modules (most deep learning frameworks, niche libraries, etc.).
• Portability matters - you want to reproduce the same environment on your laptop or another cluster.

Trade-off: conda packages are pre-compiled generic binaries, not optimized for our specific hardware. For most workloads this makes no noticeable difference, but for code that is heavily dependent on linear algebra (numpy, scipy), the pre-installed system modules can be faster because they are compiled with optimized libraries (MKL, OpenBLAS) tuned for our processors.

Approach 2: Pre-installed system modules (Python + bundles)¶

Our software stack includes optimized Python installations compiled specifically for the cluster hardware. These come with two companion bundles that provide commonly used packages:

• SciPy-bundle: scientific core packages (numpy, scipy, pandas, scikit-learn, etc.), compiled against optimized BLAS/LAPACK libraries.
• Python-bundle-PyPI: a broad collection of general-purpose packages from PyPI (requests, Jinja2, click, pytest, cryptography, and many more).

To use them:

module load SciPy-bundle/2024.05-gfbf-2024a

This automatically loads the corresponding Python version and all dependencies. You can inspect what is included with:

module show SciPy-bundle/2024.05-gfbf-2024a

When to use it:

• Your workflow depends on other system modules (e.g. a simulation code built with a specific toolchain) and you need Python packages that are compatible with them.
• You want optimized numerical performance out of the box (MKL or OpenBLAS linked numpy/scipy).
• You need MPI-aware Python packages like mpi4py that must be compiled against the cluster's MPI installation (see MPI for Python).

Trade-off: you are tied to the toolchain and versions we provide. You cannot easily install additional packages into these module-based environments. If you need extra packages on top of a system Python module, you can create a lightweight venv that inherits the module's packages:

module load SciPy-bundle/2024.05-gfbf-2024a
python -m venv --system-site-packages /scratch/$USER/my-venv
source /scratch/$USER/my-venv/bin/activate
pip install some-extra-package

The --system-site-packages flag gives the venv access to all packages from the loaded modules, so you only need to install what is missing.

Configuring conda¶

You can customize conda's behavior through the ~/.condarc file:

# Where conda looks for environments
envs_dirs:
  - /scratch/<username>/conda-envs

# Where conda caches downloaded packages
pkgs_dirs:
  - /scratch/<username>/conda-pkgs

# Show environment name in the prompt
changeps1: true
env_prompt: '({name}) '

# Default channels
channels:
  - conda-forge

See also¶

Additional resources¶

• Conda user guide
• Conda-forge package search
• Mamba documentation
• Miniforge repository