Known issues¶

Known issues affecting the DIPC HPC clusters. If you encounter a problem not listed here, contact support-hpc@dipc.org.

Ongoing¶

There are no ongoing issues at this time.

Mitigated¶

May 6, 2026¶

MITIGATED: Hyperion compute nodes drained for urgent maintenance

Update: Since Monday, May 4, we have been progressively applying the required measures to ensure the correct and secure operation of the compute nodes, on all nodes we have been able to reboot, that is, those whose running jobs had already finished. Almost all nodes have now been returned to the queues, with only a few still pending, so the cluster is operating close to full capacity. The remaining nodes will be brought back into service over the coming days as their jobs complete and we are able to reboot them. Login nodes remain fully operational, and job submission continues to work normally.

April 30, 2026

Due to urgent maintenance required on the Hyperion cluster, all compute nodes are being drained. The maintenance is critical and cannot be postponed.

What this means:

• Running jobs are not terminated. Compute nodes drain progressively as jobs finish naturally; no new work will start on a drained node.
• New submissions are allowed, but they will remain PENDING in the queue until the maintenance is complete.
• Login nodes remain fully operational. You can continue using them for data transfer, preprocessing, scripting, and other non-compute tasks.

Compute access will be restored progressively starting Monday, May 4, 2026 at the earliest, as nodes are validated and returned to service. We cannot yet confirm a full completion date; the cluster will remain partially drained until all nodes are back online. We will send a follow-up message once normal capacity is fully restored.

We apologize for the very short notice and for the disruption. This action is necessary to ensure the stability and security of the cluster. For urgent concerns, contact support-hpc@dipc.org.

Status: Mitigated. Almost all nodes are back in service; the cluster is operating close to full capacity, with a few nodes still pending reboot.

April 2, 2026¶

MITIGATED: Atlas FDR, /scratch storage target degraded (atlas-sto06)

Update: The root cause of the XFS shutdowns on atlas-sto06 could not be definitively identified, but the affected storage target has been placed in BeeGFS emergency pool mode to prevent further incidents. The /scratch filesystem is fully accessible again and Atlas FDR is operational. As a trade-off, approximately 10 TB of capacity on target 5302 are no longer available for new allocations. Given that Atlas FDR is approaching end of life, this mitigation will remain in place until the cluster is decommissioned.

March 24, 2026

On March 23, 2026, atlas-sto06 (backing BeeGFS storage target 5302 on /dev/sda1) experienced three XFS shutdowns at 14:13, 16:42 and 17:32. All three crashes were triggered by BeeGFS worker threads attempting to create a new file or directory, and all three converged on the exact same code path inside XFS:

xfs_vn_mknod -> xfs_create -> xfs_trans_cancel -> xfs_force_shutdown(0x8)

The shutdown flag 0x8 corresponds to SHUTDOWN_CORRUPT_INCORE ("corrupt in-memory data structures"), not to an I/O or device-path error. Read and write operations against existing files were unaffected; only inode allocation for new objects triggered the fault.

Diagnostics performed

To narrow down the cause we ran a full set of checks on atlas-sto06:

• RAID consistency check: completed with optimal status, no disk-level errors.
• XFS free-space fragmentation analysis: average free extent size of ~850 blocks (~3.4 MB) globally, and healthy distributions across all 32 allocation groups. No fragmentation pressure.
• Inode availability: df -i reported only ~10 M inodes used out of ~1.56 G available (≈1%). No inode exhaustion.
• xfs_repair -n (dry run): clean except for a single suspicious inode (26850315777) with a bad magic number, bad version and bad format. Minor counter drift on agf_freeblks/sb_fdblocks (off by 1). Not sufficient to explain three independent crashes on a healthy disk.
• Memtest86+ 8.00: full pass on the node's 64 GB of RAM, no errors. Hardware memory corruption ruled out.

No single check produced a definitive smoking gun, but historical reports of XFS bugs affecting file and directory creation on this kernel branch (SLES 3.0.101) are consistent with what we observed.

Mitigation applied

Since all three crashes shared the same trigger (new inode allocation via xfs_create), preventing BeeGFS from placing new chunks on the affected target is sufficient to avoid the crash path entirely, while keeping all existing data readable and writable. BeeGFS provides exactly this mechanism through the free_space.override file:

echo 0 > /beedata/stor1/free_space.override

With this override in place, BeeGFS reports target 5302 as full and moves it to the emergency pool. New chunks are allocated on the remaining healthy targets, and the xfs_create path on atlas-sto06:/dev/sda1 is no longer exercised.

Current beegfs-df state on Atlas FDR:

STORAGE TARGETS:
TargetID       Pool       Total        Free   %     ITotal      IFree   %
========       ====       =====        ====   =     ======      =====   =
   1434     normal  14891.7GiB   6939.5GiB 47%    1489.4M    1479.7M 99%
   5302  emergency  14891.7GiB      0.0GiB  0%    1489.4M    1479.7M 99%
  15491     normal  14891.7GiB   6558.2GiB 44%    1489.4M    1479.7M 99%
  19201     normal  14891.7GiB   6441.2GiB 43%    1489.4M    1479.7M 99%
  37621     normal  14891.7GiB   6324.7GiB 42%    1489.4M    1479.6M 99%
  54637     normal  14891.7GiB   7059.1GiB 47%    1489.4M    1479.6M 99%

Status: Mitigated. No new crashes since the override was applied.

Resolved¶

June 11, 2026¶

RESOLVED: Atlas FDR InfiniBand network issue, nodes preventively set to drain

Update: After analyzing the InfiniBand fabric logs and monitoring data, the source of the instability was identified and the corresponding corrective measures were applied. The fabric has since stabilized, with no intervention at the IB switch level required and no interruption to running jobs or IB-based mounts. After a monitoring period confirming stable operation, the Atlas FDR nodes were returned to service and have been active since June 10, 2026.

June 9, 2026

An issue with the cluster's InfiniBand network is under investigation. For now the service remains operational and running jobs continue normally.

As a precaution, the Atlas FDR nodes have been placed in DRAIN mode: running jobs will continue, but no new jobs will start until the situation is clarified.

Depending on the diagnosis, action at the IB switch level may be required. If that becomes necessary, there could be a temporary InfiniBand network interruption affecting active jobs and IB-based mounts. A follow-up notification with specific details would be sent in advance in that case.

No user action is required at this time. Further updates will follow as the investigation progresses.

Status: Resolved. The fabric is stable and all Atlas FDR nodes are back in service since June 10, 2026.

May 25, 2026¶

RESOLVED: Hyperion accessibility issues during the weekend

Update: Access to Hyperion was temporarily unavailable during the weekend due to a data center overheating issue, which triggered an automatic safety shutdown of the /home filesystem nodes. Following service restoration, the authentication system on the management nodes temporarily failed, preventing users from submitting jobs, querying the queues, and performing other SLURM operations. This was also resolved shortly afterwards. All systems are now running normally. While we might need to shut down some nodes later if temperatures rise again, all nodes will remain online for the time being.

Status: Resolved. All systems running normally; temperatures are being monitored.

March 27, 2026¶

RESOLVED: Hyperion Cluster, Cooling System Failure

Update: The maintenance provider has successfully repaired the cooling system. Both racks have been powered back on and all nodes are now operational.

January 23, 2025

A failure in one of the external cooling compressors for the Hyperion cluster occurred following electrical maintenance, forcing the shutdown of two full racks. Almost 50 compute nodes will remain offline until the maintenance provider responsible for the cooling system identifies and resolves the issue.

Here's the list of the affected nodes: