Python¶

Python is an interpreted, high-level, general-purpose programming language, predominantly utilized for scientific computing.

Default Python Version¶

By default, all HPC systems use the Anaconda Python Distribution. Anaconda is a free, open-source distribution of Python that simplifies package management and deployment without compromising performance.

Python 3 is the default version on our systems. To access it, load Python with the following command:

$ module load Python

You can view all available Python versions with:

module spider Python

Upon loading a Python module, you will be using the base environment. While the base environment might suffice for many users, if you require additional packages, the DIPC staff can install them upon request. However, we encourage users to create their own conda environments where they can manage their packages independently.

We also offer preinstalled environments that you can list with the following command:

$ conda env list

The naming convention for environments follows the main_package-version-build format.

Managing Conda Environments¶

This section details how to manage your Python packages via conda in your HPC account. This process requires no root privileges, enabling you to customize Python to your needs without interfering with any system-level Python installations or other libraries or tools.

Creating a Conda Environment¶

Command-Based Environment Creation¶

First, you need to load Python to your environment:

$ module load Python

To create a conda environment, use:

$ conda create --name myenv

You can specify a version of Python or a specific package while creating an environment:

conda create --name myenv python=3.6
conda create -n myenv scipy
conda create -n myenv python=3.7 scipy=1.4.1 astroid babel

Creating an Environment in a Custom Location¶

Environments are typically created in your home directory, which is not readable from compute nodes. To overcome this, install your environment under your /scratch directory:

conda create --prefix /scratch/<username>/conda-env/<my conda env name>

Activate the environment by providing the path to the installation:

conda activate /scratch/<username>/conda-env/<my conda env name>

Creating an Environment from a File¶

You can create environments using configuration files that specify all required packages:

$ conda env create -f myconfigfile.yml

Here's an example YAML configuration file for creating a conda environment with TensorFlow 2.1.0:

name: tensorflow-2.1.0-gpu_py37h7a4bb67_0
channels:
  - defaults
dependencies:
  - cudatoolkit=10.1.243=h6bb024c_0
  - cudnn=7.6.5=cuda10.1_0
  - keras-applications=1.0.8=py_0
  - keras-preprocessing=1.1.0=py_0
  - more-itertools=8.2.0=py_0
  - matplotlib-base=3.1.3=py37hef1b27d_0
  - matplotlib=3.1.3=py37_0
  - mkl=2020.1=217
  - mkl-dnn=0.19=hfd86e86_1
  - mkl-include=2020.1=217
  - mkl-service=2.3.0=py37he904b0f_0
  - mkl_fft=1.0.6=py37hd81dba3_0
  - numpy=1.18.1=py37h4f9e942_0
  - numpy-base=1.18.1=py37hde5b4d6_0
  - pandas=1.0.3=py37h0573a6f_0
  - scipy=1.4.1=py37h0b6359f_0
  - tensorflow=2.1.0=gpu_py37h7a4bb67_0

Activating and Utilizing a Conda Environment¶

After successfully creating a conda environment, it is ready to be activated and utilized. The command is as follows:

$ conda activate myenv

Customizing Your Conda Environments¶

Advanced configuration options, such as modifying the appearance of the prompt or defining the path where conda searches for environments, can be modified in the conda.rc file located in your home directory.

# Directories where conda looks for environments
envs_dirs:
  - /scratch/username/conda/env
# Select if prompt changes when a environment is activated
changeps1: true
# Select how the prompt looks like after activating a environment
env_prompt: '({name}) '

The conda.rc file can be manually edited or modified via the conda config command line option.

Installing Packages into an Existing conda Environment¶

You can find packages easily using the following command:

$ conda search numpy

After identifying the most suitable package, you can install it using this command:

$ conda install numpy=1.18.1=py37h94c655d_0

Ensure the Python versions are compatible.

If the conda search fails to locate the desired package, you can install it using pip within your environment.

You might also want to consider exploring alternative channels if necessary.

Installing, Updating, and Removing Python Packages Using Pip¶

While Conda is a powerful package manager designed specifically for scientific Python, not every Python package is available through Conda. For these cases, Python's native package manager pip can be used.

Installing Packages¶

In the event the conda search command fails to locate the package you require, you can install it using pip. First, ensure that pip is installed in your conda environment. In order to do that it is usually a good idea to create the conda environment with a specific version of Python.

$ conda create --prefix /scratch/<username>/conda-env/<my conda env name> python=3.10

This will install pip along with the specified Python version in the conda environment.

You can install the packages with:

$ pip install package

Updating Packages¶

To update a package using pip, you can use the install command with the --upgrade option:

$ pip install --upgrade package

Removing Packages¶

To remove a package installed with pip, use the uninstall command:

$ pip uninstall package

Please confirm the removal when prompted.

Remember, while pip can be a useful tool for managing Python packages, it is recommended to use conda whenever possible when working within a conda environment. This is because conda is aware of the environment's dependencies, which can help prevent conflicts and ensure smooth operation of your Python software.

Replicating a conda Environment¶

You can easily replicate a conda environment by cloning it:

$ conda create --name myclone --clone myenv

Different Python Versions¶

Each compiler version installed on the system is linked to a Python version. These modules are hidden by default, but can be listed as follows:

$ module --show-hidden spider Python

Likewise, each Python version has an associated module named SciPy-Bundle that loads a group of commonly used packages:

$ module spider SciPy-Bundle

By loading this module, you gain access to packages such as: numpy, scipy, scikit-learn, and more.

Additional Resources¶

• Managing conda environments - Official documentation