Your first Python environment¶

This guide walks you through creating, using, and managing a Python environment on the clusters. By the end you will have a working conda environment, know how to use it in SLURM jobs, and understand how to keep it safe from scratch cleanups.

Prerequisites

You should be comfortable connecting to the cluster and submitting jobs. If not, start with the Quick Start and Your first job tutorials first.

1. Load Miniforge3¶

All Python work on our clusters starts by loading the Miniforge3 module:

module load Miniforge3

This activates a minimal base environment called miniforge3-base and gives you access to python, conda, mamba, and pip:

==============================================================================
                        Miniforge3 Environment Loaded
==============================================================================
  Activated environment: miniforge3-base

  NOTE: This is a custom minimal environment.
        We recommend creating your own conda environments for your
        projects.
==============================================================================

Before creating or installing anything, deactivate this base environment. This ensures your new environment will be clean and will not inherit packages from the system base:

conda deactivate

What is mamba?

mamba is a drop-in replacement for conda that resolves dependencies much faster. Use mamba wherever you would use conda (except for activate/deactivate, which always use conda).

2. Create an environment¶

Create a new environment in /scratch with the packages you need:

mamba create --prefix /scratch/$USER/conda-envs/myproject python=3.12 numpy scipy matplotlib

This will:

Create a directory at /scratch/$USER/conda-envs/myproject
Install Python 3.12 and the three requested packages (plus their dependencies)
Download everything from the conda-forge channel

Where to create environments

Atlas EDR and Atlas FDR: your home directory (/dipc) is not accessible from compute nodes. Environments installed there will not work in SLURM jobs. Always use /scratch.
Hyperion: you can also create environments in /data (from login nodes), which provides more permanent storage. However, /data is read-only on compute nodes, so environments there must be fully built before submitting jobs.

Why --prefix instead of --name?

The Miniforge3 module is a system-wide installation where you don't have write permissions. Using --name would try to create the environment inside that installation, which will fail. --prefix lets you choose a writable location. See the Python reference for how to configure conda so --name works with your own directory.

3. Activate and use it¶

conda activate /scratch/$USER/conda-envs/myproject

Your prompt will change to show the environment name. Now you can run Python with all the packages you installed:

python -c "import numpy as np; print(f'numpy {np.__version__} loaded successfully')"

To leave the environment:

conda deactivate

4. Install additional packages¶

Need more packages? Activate the environment first, then install:

conda activate /scratch/$USER/conda-envs/myproject
mamba install pandas scikit-learn

For packages not available on conda-forge, use pip:

pip install some-package

conda first, pip second

Always install as many packages as possible with mamba/conda first, then use pip only for what is not available on conda-forge. Mixing the two carelessly can cause dependency conflicts.

5. Use it in a SLURM job¶

To use your environment in a batch job, load the module and activate the environment in your script:

my_python_job.slurm

#!/bin/bash
#SBATCH --job-name=python-job
#SBATCH --partition=general
#SBATCH --qos=test
#SBATCH --nodes=1
#SBATCH --ntasks=1
#SBATCH --cpus-per-task=1
#SBATCH --mem=4gb
#SBATCH --time=00:10:00
#SBATCH --output=%x.%j.out
#SBATCH --error=%x.%j.err

module load Miniforge3
conda activate /scratch/$USER/conda-envs/myproject

python my_analysis.py

Submit it:

cd /scratch/$USER
sbatch my_python_job.slurm

6. Protect your environment¶

Environments in /scratch can be lost during storage cleanups. Save yours to a YAML file so you can recreate it at any time:

conda activate /scratch/$USER/conda-envs/myproject
conda env export --no-builds > myproject-env.yml

Keep this file in your home directory (which is backed up). If you ever lose the environment, recreate it with:

module load Miniforge3
mamba env create -f myproject-env.yml --prefix /scratch/$USER/conda-envs/myproject

Tip

Make it a habit to re-export your environment file every time you install new packages. A backup that is out of date is almost as bad as no backup.

Common problems¶

My environment takes forever to create

Use mamba instead of conda. It is significantly faster at resolving dependencies.

conda activate says the environment does not exist

Make sure you are using the full path: conda activate /scratch/$USER/conda-envs/myproject, not just the name. Alternatively, configure envs_dirs in your ~/.condarc (see Configuring conda).

My job fails with ModuleNotFoundError

The most common cause is forgetting to activate the environment in your SLURM script. Make sure both module load Miniforge3 and conda activate ... are in the script before your python command.

My home directory is full

Conda caches downloaded packages in ~/.conda/pkgs by default. Move the cache to /scratch by adding this to your ~/.condarc:

~/.condarc

pkgs_dirs:
  - /scratch/<username>/conda-pkgs

Then clean the old cache: conda clean --all

I need packages from the system modules (MKL-optimized numpy, mpi4py, etc.)

Instead of conda, create a lightweight venv on top of system modules:

module load SciPy-bundle/2024.05-gfbf-2024a
python -m venv --system-site-packages /scratch/$USER/my-venv
source /scratch/$USER/my-venv/bin/activate
pip install some-extra-package

See Python: Two approaches for details.

What's next¶

Python reference

Full reference: all conda options, system modules, pip, bioconda, and configuration.

Python
Environment modules

Load pre-installed software, compilers, and libraries.

Modules
Your first job

Learn the basics of SLURM job submission.

First job
Software catalog

Check what is already installed before creating your own environment.

Catalog