Atlas EDR system¶

How to log into Atlas EDR¶

To establish a connection a SSH client is necessary. Please, refer to SSH for more information.

Establish connection Atlas-EDR:

$ ssh username@atlas-edr.sw.ehu.es

You can also establish direct connection with the login nodes:

$ ssh username@atlas-edr-login-01.sw.ehu.es
$ ssh username@atlas-edr-login-02.sw.ehu.es

You would need to bring your files and data over, compile your code or use the compiled one, and create a batch submission script. Then submit that script so that your application runs on the compute nodes. Pay attention to the various file systems available and the choices in programming environments.

Specifications¶

Atlas EDR employs Infiniband EDR technology for the interconnection network. Network topology is a fat-tree with a 5:1 blocking factor.

Filesystems and IO¶

Login Nodes¶

• Atlas-EDR has 2 login nodes: atlas-edr-login-01.sw.ehu.es and atlas-edr-login-02.sw.ehu.es.
• Each node has two sockets populated with a 48 core Intel Xeon Platinum 8260 each.
• Each node has 64 GB of RAM.

Job submission¶

Here you will find some batch scripts you can use as template to submit your jobs. For more specific information about how to submit jobs please visit SLURM resource manager webpage.

QoS and partitions¶

Users can request a partition for each job they submit. These are the available partitions:

^{*(D) = Default partition}

On Atlas-EDR, we conceptualize a partition as a set of nodes to which we can associate a Quality of Service (QoS). As such, we only have two partitions, being one the general partition that encompasses all nodes for public or general use, and the preemption partition, which exhibits distinct behavior that we will elaborate on later. This preemption partition includes all nodes exclusively designated for use by their respective ownership groups.

Having explained this, users must select one of the following QoSs when submitting a job:

^{*(D) = Default QoS}

This is what each columns means:

• MaxWall: Maximum amount of time the job is allowed to run. 1-00:00:00 reads as one day or 24 hours.
• MaxNodesPU: Maximum amount of nodes user's jobs can use at a given time.
• MaxJobsPU: Maximum number of running jobs per user.
• MaxSubmitPU: Maximum number of jobs that can be submitted to the QoS/partition.
• MaxTRES: Maximum amount of trackable resources.

srun¶

Using srun in your batch scripts simplifies jobs execution. srun will directly gather the information provided in the resource specification list and will allocate the resources for the job automatically. Some example of batch script using srun are provided bellow.

Basic submission script for MPI applications¶

#!/bin/bash
#SBATCH --qos=regular
#SBATCH --job-name=JOB_NAME
#SBATCH --cpus-per-task=1
#SBATCH --mem=200gb
#SBATCH --nodes=8
#SBATCH --ntasks-per-node=48

module load program/program_version

srun binary < input 

#!/bin/bash
#SBATCH --qos=regular
#SBATCH --job-name=JOB_NAME
#SBATCH --cpus-per-task=1
#SBATCH --mem=200gb
#SBATCH --nodes=8
#SBATCH --ntasks-per-node=48

module load program/program_version

mpirun -np $SLURM_NTASKS binary < input 

Basic submission script for OpenMP applications¶

For a OpenMP application the number of threads can be controlled defining the OMP_NUM_THREADS or SLURM's --cpus-per-task job directive. If this variable is not defined, the number of threads created will be equal to the amount of cores reserved in your cpuset, that is, the number of cores requested in the batch script.

#!/bin/bash
#SBATCH --qos=regular
#SBATCH --job-name=JOB_NAME
#SBATCH --cpus-per-task=48
#SBATCH --mem=200gb
#SBATCH --nodes=1
#SBATCH --ntasks-per-node=1

module load program/program_version

srun binary < input

#!/bin/bash
#SBATCH --qos=regular
#SBATCH --job-name=JOB_NAME
#SBATCH --cpus-per-task=48
#SBATCH --mem=200gb
#SBATCH --nodes=1
#SBATCH --ntasks-per-node=1

module load program/program_version

export OMP_NUM_THREADS=$SLURM_CPUS_PER_TASK

binary < input

Basic submission script for Hybrid (MPI+OpenMP) applications¶

#!/bin/bash
#SBATCH --qos=regular
#SBATCH --job-name=JOB_NAME
#SBATCH --cpus-per-task=4
#SBATCH --mem=200gb
#SBATCH --nodes=2
#SBATCH --ntasks-per-node=12

module load program/program_version

export OMP_NUM_THREADS=$SLURM_CPUS_PER_TASK
mpirun -np $SLURM_NTASKS binaryi < input

#!/bin/bash
#SBATCH --qos=regular
#SBATCH --job-name=JOB_NAME
#SBATCH --cpus-per-task=4
#SBATCH --mem=200gb
#SBATCH --nodes=2
#SBATCH --ntasks-per-node=12

module load program/program_version

export OMP_NUM_THREADS=$SLURM_CPUS_PER_TASK
srun -n $SLURM_NTASKS binary

Basic submission script for GPGPU capable applications¶

#!/bin/bash
#SBATCH --qos=regular
#SBATCH --job-name=JOB_NAME
#SBATCH --cpus-per-task=1
#SBATCH --gres=gpu:p40:1
#SBATCH --mem=200gb
#SBATCH --nodes=1
#SBATCH --ntasks-per-node=1

module load program/program_version

srun binary < input

In general you can request up to 2 GPUs per node. In order to do that you can adjust the corresponding line in the batch script:

#SBATCH --gres=gpu:p40:2

Running jobs on GPUs¶

There are two different types of GPUs on Atlas EDR:

This sums it up:

Software¶

Compiling your code¶

Intel compilers are recommended for building your applications on Atlas-EDR. There is no system default modulefile that takes care of this. Use the module avail command to see what versions are available and load an Intel compiler module before compiling. For example:

$ module load intel/2020a

Notice that when a compiler module is loaded, some environment variables are set or modified to add the paths to certain commands, include files, or libraries, to your environment. This helps to simplify the way you do your work.

As an alternative, Atlas-EDR also offers a collection of open source tools such as compilers or scientific libraries. Use module avail command to see versions available. For example:

module load intel/2020a
module load FFTW/3.3.8-intel-2020a