Atlas EDR¶
Atlas EDR is a DIPC cluster based on Intel Xeon Skylake and Cascade Lake processors with NVIDIA GPUs (Tesla P40 and RTX 3090). It uses an Infiniband EDR interconnect in a fat-tree topology with a 5:1 blocking factor.
Atlas EDR and Atlas FDR share a single SLURM instance (same partitions and QoS limits) but have separate login nodes, compute nodes, /scratch filesystems, and software stacks. Jobs submitted from Atlas EDR login nodes run exclusively on EDR compute nodes.
-
2,656 cores
85 compute nodes with Skylake and Cascade Lake processors.
-
54 GPUs
NVIDIA Tesla P40, RTX 3090, RTX 8000, and A30 accelerators.
-
16 TB RAM
From 64 GB to 2 TB per node.
-
EDR InfiniBand
100 Gb/s interconnect.
Connecting and login nodes¶
ssh <username>@atlas-edr.sw.ehu.es
You can also connect directly to a specific login node:
ssh <username>@atlas-edr-login-01.sw.ehu.es
ssh <username>@atlas-edr-login-02.sw.ehu.es
| Login node | Hostname | Processor | Cores | Memory |
|---|---|---|---|---|
| atlas-edr-login-01 | atlas-edr-login-01.sw.ehu.es |
2x Intel Xeon Platinum 8260 (Cascade Lake) | 96 | 64 GB |
| atlas-edr-login-02 | atlas-edr-login-02.sw.ehu.es |
2x Intel Xeon Platinum 8260 (Cascade Lake) | 96 | 64 GB |
Warning
Login nodes are for file management, editing, and compilation only. Do not run computations on login nodes. If a resource-intensive process is detected, all your processes will be terminated and your account will be suspended until you contact support-hpc@dipc.org.
For detailed SSH configuration, key-based authentication, and troubleshooting, see Connecting via SSH.
Specifications¶
Compute nodes |
Count |
Processor |
Cores |
Memory |
Accelerator |
|---|---|---|---|---|---|
| atlas-[249-256] | 8 | Intel Xeon Platinum 8168 | 48 | 384 GB | 2x NVIDIA Tesla P40 |
| atlas-[257-280] | 24 | Intel Xeon Platinum 8168 | 48 | 384 GB | 1x NVIDIA Tesla P40 |
| atlas-[281-283] | 3 | Intel Xeon Platinum 8168 | 48 | 64 GB | - |
| atlas-[284-285] | 2 | Intel Xeon Platinum 8168 | 48 | 384 GB | - |
| atlas-[286-293] | 8 | Intel Xeon Platinum 8280 | 56 | 192 GB | - |
| atlas-295 | 1 | Intel Xeon Platinum 8268 | 48 | 384 GB | - |
| atlas-[296-298] | 3 | Intel Xeon Platinum 8268 | 48 | 96 GB | - |
| atlas-[301-303] | 3 | Intel Xeon Platinum 8268 | 48 | 96 GB | - |
| atlas-[304-327] | 24 | Intel Xeon Gold 6248R | 48 | 192 GB | - |
| atlas-[328-334] | 7 | Intel Xeon Gold 6248R | 48 | 96 GB | 2x NVIDIA RTX 3090 |
| atlas-[364-365] | 2 | Intel Xeon Gold 6248R | 48 | 384 GB | - |
Compute nodes |
Count |
Processor |
Cores |
Memory |
Accelerator |
|---|---|---|---|---|---|
| atlas-294 | 1 | Intel Xeon Platinum 8268 | 48 | 1280 GB | - |
| atlas-[336-348,350-355] | 19 | Intel Xeon Platinum 8168 | 48 | 384 GB | - |
| atlas-349 | 1 | Intel Xeon Platinum 8168 | 48 | 384 GB | 1x NVIDIA RTX 8000 |
| atlas-[356-361] | 6 | Intel Xeon Gold 6248R | 48 | 96 GB | - |
| atlas-362 | 1 | Intel Xeon Gold 6248R | 48 | 96 GB | 1x NVIDIA A30 |
| atlas-363 | 1 | Intel Xeon Platinum 8380 | 80 | 1024 GB | - |
| atlas-367 | 1 | Intel Xeon Gold 6248R | 48 | 512 GB | 1x NVIDIA A30 |
| atlas-368 | 1 | Intel Xeon Platinum 8362 | 64 | 2048 GB | - |
Filesystems¶
| Filesystem | Mount point | Quota | Purpose | Compute nodes | Backup |
|---|---|---|---|---|---|
| Home | /dipc |
None | Permanent storage, configuration files | Not accessible (symlink to /scratch) |
Daily (up to 2 months) |
| Scratch | /scratch |
1.5 TB / 3M files | Job working directory, I/O | Read-write | No |
| Lscratch | /lscratch |
None | Fast local I/O for single-node jobs | Read-write (local disk, ~750 GB) | No |
Checking your quota¶
getquota
/dipc (home)¶
Your home directory (/dipc/<username>) is where you land when you log in. It runs on a Ceph parallel filesystem (~800 TB total) and currently has no per-user quota, though we ask users to keep usage reasonable. It is shared across all DIPC HPC systems.
Daily backups are performed and retained for up to 2 months. If you lose data, contact support-hpc@dipc.org and we will help recover it.
/dipc is not accessible from compute nodes
On Atlas EDR, /dipc is only mounted on the login nodes. On compute nodes, /dipc is a symbolic link pointing to /scratch. This means you cannot use your home directory as a working directory for jobs. Always submit jobs from and direct output to /scratch.
Tip
Home directories are private by default. You can share access using chmod or POSIX ACLs.
/scratch¶
/scratch is a high-performance shared filesystem for job working directories. A personal directory /scratch/<username> is created automatically with your account.
All job I/O should go through /scratch. Submit your jobs from here and direct output here.
Not permanent storage
/scratch is not for long-term storage. When occupancy exceeds 80%, performance degrades for all users. Move results you want to keep to /dipc and clean up finished jobs promptly.
Avoid many small files
The filesystem performs poorly with high volumes of small files. Avoid having more than a few thousand files in a single directory. The per-user file limit is 3,000,000.
/lscratch¶
/lscratch is a node-local filesystem (~750 GB per node). It is faster than /scratch for single-node jobs with heavy local I/O, but the data is only visible to the node where the job runs.
Warning
No user directory is created automatically in /lscratch. Create your own at the start of your job and clean it up when done:
mkdir -p /lscratch/$USER/$SLURM_JOB_ID
# ... use it for I/O ...
rm -rf /lscratch/$USER/$SLURM_JOB_ID
Running jobs¶
-
SLURM workload manager
Partitions, QoS limits, batch script templates, and all SLURM commands.
-
GPU accelerators
GPU inventory, SLURM request syntax, and CUDA compatibility.
-
Your first job
Step-by-step tutorial to get started with job submission.
Software¶
Atlas EDR provides scientific software through environment modules.
module avail # list all available modules
module spider VASP # search for a specific package
module load intel/2020a # load a specific module
For details on compilers, optimization flags, and available toolchains, see Compilers.