Hyperion¶
Hyperion is the main DIPC cluster, featuring Intel Xeon processors across three microarchitectures (Cascadelake, Icelake, and Emerald Rapids), NVIDIA GPUs, and an Infiniband HDR interconnect (200 Gb/s per direction).
-
17,292 cores
302 compute nodes across Intel Cascadelake, Icelake, and Emerald Rapids, and AMD Zen 3 architectures.
-
105 GPUs
NVIDIA A100, RTX 3090, RTX 4090, A6000, A30 and RTX PRO 6000 accelerators.
-
222 TB RAM
From 96 GB to 4 TB per node.
-
HDR InfiniBand
200 Gb/s interconnect.
Connecting and login nodes¶
ssh <username>@hyperion.sw.ehu.es
You can also connect directly to a specific login node:
ssh <username>@hyperion-01.sw.ehu.es
ssh <username>@hyperion-02.sw.ehu.es
| Login node | Hostname | Processor | Cores | Memory |
|---|---|---|---|---|
| hyperion-01 | hyperion-01.sw.ehu.es |
2x Intel Xeon Platinum 8362 (Icelake) | 96 | 256 GB |
| hyperion-02 | hyperion-02.sw.ehu.es |
2x Intel Xeon Platinum 8362 (Icelake) | 96 | 256 GB |
Warning
Login nodes are for file management, editing, and compilation only. Do not run computations on login nodes. If a resource-intensive process is detected, all your processes will be terminated and your account will be suspended until you contact support-hpc@dipc.org.
For detailed SSH configuration, key-based authentication, and troubleshooting, see Connecting via SSH.
Specifications¶
Compute nodes |
Processor |
Cores |
Memory |
Accelerator |
Count |
|---|---|---|---|---|---|
| hyperion-[001-005,023-029,045-051,067-073,089-095,111-117,133-139,155-161,177-181,206] | Intel Xeon Gold 6342 (Icelake) | 48 | 256 GB | - | 58 |
| hyperion-[030,052,074,096,118,140,162,182] | Intel Xeon Gold 6248R (Cascadelake) | 48 | 96 GB | 2x NVIDIA RTX 3090 24 GB | 8 |
| hyperion-[011-022,031-044,053-066,075-088,097-110,119-132,141-150,163-175] | Intel Xeon Gold 6248R (Cascadelake) | 48 | 96 GB | - | 105 |
| hyperion-[208-224,234-251,263] | Intel Xeon Platinum 8362 (Icelake) | 64 | 2 TB | - | 36 |
| hyperion-252 | Intel Xeon Gold 6348 (Icelake) | 56 | 1 TB | 8x NVIDIA A100 PCIe 80 GB | 1 |
| hyperion-253 | Intel Xeon Gold 6348 (Icelake) | 56 | 1 TB | 8x NVIDIA A100 SXM4 80 GB | 1 |
| hyperion-[254,257] | Intel Xeon Platinum 8358 (Icelake) | 64 | 2 TB | 8x NVIDIA A100 PCIe 80 GB | 2 |
| hyperion-262 | Intel Xeon Platinum 8362 (Icelake) | 64 | 2 TB | 1x NVIDIA RTX A6000 48 GB | 1 |
| hyperion-[282-284] | Intel Xeon Platinum 8358 (Icelake) | 64 | 2 TB | 8x NVIDIA A100 SXM4 80 GB | 3 |
| hyperion-[285-292] | Intel Xeon Platinum 8570 (Emerald Rapids) | 112 | 2 TB | - | 8 |
| hyperion-[297-303] | Intel Xeon 6760P (Emerald Rapids) | 128 | 2 TB | - | 7 |
Compute nodes |
Processor |
Cores |
Memory |
Accelerator |
Count |
|---|---|---|---|---|---|
| hyperion-[176,180-181,183-205,207] | Intel Xeon Gold 6248R (Cascadelake) | 48 | 192 GB | - | 27 |
| hyperion-[225-233] | Intel Xeon Platinum 8368 (Icelake) | 76 | 2 TB | - | 9 |
| hyperion-[255-256] | AMD EPYC 75F3 (Zen 3) | 64 | 1 TB | 8x NVIDIA A100 SXM4 80 GB | 2 |
| hyperion-[258-259] | Intel Xeon Platinum 8362 (Icelake) | 64 | 4 TB | - | 2 |
| hyperion-[260-261] | Intel Xeon Gold 6348 (Icelake) | 56 | 1.5 TB | 1x NVIDIA A30 24 GB | 2 |
| hyperion-[264-280] | Intel Xeon Platinum 8362 (Icelake) | 64 | 2 TB | - | 17 |
| hyperion-281 | Intel Xeon Platinum 8362 (Icelake) | 32 | 512 GB | - | 1 |
| hyperion-294 | Intel Xeon Platinum 8568Y+ (Emerald Rapids) | 96 | 512 GB | - | 1 |
| hyperion-295 | Intel Xeon Gold 5418N (Icelake) | 48 | 512 GB | - | 1 |
| hyperion-296 | Intel Xeon Platinum 8592+ (Emerald Rapids) | 128 | 512 GB | - | 1 |
| hyperion-304 | AMD EPYC 9474F (Zen 3) | 48 | 384 GB | 1x NVIDIA RTX PRO 6000 96GB | 1 |
| hyperion-305 | AMD EPYC 7773X (Zen 3) | 128 | 1 TB | 7x NVIDIA RTX 3090 24GB 1x NVIDIA RTX 4090 24GB | 1 |
| hyperion-306 | Intel Xeon Platinum 8380 | 80 | 1TB | - | 1 |
Private nodes with non-standard architectures
Some nodes in the preemption partition belong to research groups and use CPU architectures (e.g. Zen 3, Zen 4) that are outside the general-purpose set (CascadeLake, IceLake, Emerald Rapids). Software for these architectures is visible through the module system, but we only compile it on demand for the owning groups; the full software stack available on general-purpose nodes is not guaranteed to be present. For details about which nodes are affected, contact support-hpc@dipc.org.
Compute nodes |
Processor |
Cores |
Memory |
Accelerator |
Count |
|---|---|---|---|---|---|
| hyperion-253 | Intel Xeon Gold 6348 (Icelake) | 56 | 1 TB | 8x NVIDIA A100 SXM4 80 GB | 1 |
| hyperion-[282-284] | Intel Xeon Platinum 8358 (Icelake) | 64 | 2 TB | 8x NVIDIA A100 SXM4 80 GB | 3 |
Check available resources with cluster-resources script¶
We have prepared a handy script called cluster-resources to help you check the available resources on the cluster at any given time:
hyperion-login-01:~$ cluster-resources
GPU Node Availability in general partition (At least 1 GPUs, 1 CPUs and 1GB memory free):
Node Partition State GPU Type FreeGPUs FreeCPUs FreeMem(GB) Microarchitecture
----------------------------------------------------------------------------------------------------------------------------
🔷 hyperion-252 general MIXED a100-pcie 2/8 32/56 15/1007 IceLake
🔷 hyperion-282 general MIXED a100-sxm4 2/8 48/64 1764/2014 IceLake
GPU Node Availability in preemption partition (At least 1 GPUs, 1 CPUs and 1GB memory free):
Node Partition State GPU Type FreeGPUs FreeCPUs FreeMem(GB) Microarchitecture
----------------------------------------------------------------------------------------------------------------------------
🔷 hyperion-256 hitz-exclusive MIXED a100-sxm4 3/8 34/64 556/1007 ZenHitz
🔷 hyperion-260 eiguren MIXED a30 1/1 20/56 972/1510 IceLake
🔷 hyperion-261 eiguren MIXED a30 1/1 24/56 1450/1510 IceLake
Icons:
✅: All resources available
🔷: At least requested resources available
The utilization of the script along with all its filtering options can be checked by running cluster-resources --help.
Filesystems¶
| Filesystem | Mount point | Quota | Purpose | Compute nodes | Backup |
|---|---|---|---|---|---|
| Home | /dipc |
50 GB / 300K files | Configuration files, dotfiles, small data | Read-write | No |
| Scratch | /scratch |
2 TB / 10M files | Job working directory, I/O | Read-write (BeeGFS over Infiniband) | No |
| Data | /data |
10 TB / 20M files | Long-term storage of results | Read-only (NFS over 100 GbE) | No |
| Lscratch | /lscratch |
None | Fast local I/O for single-node jobs | Read-write (local disk) | No |
No backups
No backups are performed for any filesystem on Hyperion. You are responsible for safeguarding your important data.
Checking your quota¶
The getquota alias has been configured, so you can check your quota usage across al the filesystems at once:
user@hyperion-login-02:~$ getquota
##### Checking quota in /home filesystem for user #####
Quota information for storage pool Default (ID: 1):
user/group || size || chunk files
name | id || used | hard || used | hard
--------------|------||------------|------------||---------|---------
user| XXXX|| 282.98 MiB| 50.00 GiB|| 3509| 300000
##### Checking quota in /scratch filesystem for user #####
Quota information for storage pool Default (ID: 1):
user/group || size || chunk files
name | id || used | hard || used | hard
--------------|------||------------|------------||---------|---------
user| XXXX|| 1.02 GiB| 2.00 TiB|| 25371| 10000000
##### Checking quota in /data filesystem for user #####
Blocks Inodes
User ID Used Soft Hard Warn/Grace Used Soft Hard Warn/Grace
---------- --------------------------------- ---------------------------------
username 0 10T 10.5T 00 [------] 1 20m 20.5m 00 [------]
getquota-home
getquota-scratch
getquota-data
/dipc (home)¶
Your home directory (/dipc/<username>) is where you land when you log in. It is shared between login and compute nodes. Use it for configuration files, dotfiles, and small data. Do not use it for job I/O or large datasets.
Tip
Home directories are private by default. You can share access using chmod or POSIX ACLs.
/scratch¶
/scratch is a high-performance parallel filesystem (BeeGFS over Infiniband) for job working directories. A personal directory /scratch/<username> is created automatically with your account.
All job I/O should go through /scratch. Submit your jobs from here and direct output here.
Not permanent storage
/scratch is not for long-term storage. When occupancy exceeds 80%, performance degrades for all users. Move results you want to keep to /data and clean up finished jobs promptly.
Avoid many small files
The filesystem performs poorly with high volumes of small files. Avoid having more than a few thousand files in a single directory. If your workflow generates many small files, consider packing them into archives (tar) or using /lscratch for local I/O.
/data¶
/data is designed for long-term storage of data that is not actively used in running jobs, such as results from completed calculations that you want to retain.
Unlike /scratch, /data is not a parallel filesystem. It is exported via NFS over the 100 GbE management network and is mounted read-only on compute nodes. You can only write to /data from login nodes.
Grace period
If you exceed the 10 TB or 20M file quota, a one-day grace period allows temporary usage of up to 10.5 TB and 20.5M files. An automatic email alert is sent when you enter the grace period.
/lscratch¶
/lscratch is a node-local filesystem (the physical disks inside each compute node). It is faster than /scratch for single-node jobs with heavy local I/O, but the data is only visible to the node where the job runs.
Warning
No user directory is created automatically in /lscratch. Create your own at the start of your job and clean it up when done:
mkdir -p /lscratch/$USER/$SLURM_JOB_ID
# ... use it for I/O ...
rm -rf /lscratch/$USER/$SLURM_JOB_ID
The size of /lscratch varies per node:
lscratch sizes by node
Compute nodes |
/lscratch size |
|---|---|
| hyperion-[001-005,023-029,045-051,067-073,089-095,111-117,133-139,155-161,177-181,206] | 1.7 TB |
| hyperion-[030,052,074,096,118,140,162,182] | 1.7 TB |
| hyperion-[011-022,031-044,053-066,075-088,097-110,119-132,141-154,163-175] | 1.5 TB |
| hyperion-[208-224,234-251] | 3.6 TB |
| hyperion-[252-253] | 1.4 TB |
| hyperion-[254,257] | 3.3 TB |
| hyperion-[262-263] | 800 GB |
| hyperion-[282-284] | 3.3 TB |
| hyperion-[285-292] | 5.7 TB |
Compute nodes |
/lscratch size |
|---|---|
| hyperion-[176,180-181,183-205,207] | 1.5 TB |
| hyperion-[225-233] | 6.9 TB |
| hyperion-[255-256] | 3.3 TB |
| hyperion-[258-259] | 7.0 TB |
| hyperion-[260-261] | 750 GB |
| hyperion-264 | 7.0 TB |
| hyperion-[265-280] | 3.5 TB |
| hyperion-281 | 800 GB |
| hyperion-294 | 14 TB |
| hyperion-295 | 3.4 TB |
Running jobs¶
-
SLURM workload manager
Partitions, QoS limits, batch script templates (MPI, OpenMP, hybrid, GPU), and all SLURM commands.
-
GPU accelerators
Full hardware inventory, SLURM request syntax, and A100 usage policy.
-
VNC remote desktop
Interactive graphical sessions on compute nodes.
-
Your first job
Step-by-step tutorial to get started with job submission.
Software¶
Hyperion provides scientific software through environment modules. All centrally installed software is compiled for Cascadelake, Icelake, and Emerald Rapids, so the same modules are available regardless of which node type you are on.
module avail # list all available modules
module spider VASP # search for a specific package
module load intel/2022a # load a specific module
For details on compilers, optimization flags, and architecture-specific targets, see Compilers.