Storage and quotas¶

Our clusters provide several filesystems, each designed for a different purpose. Understanding which one to use for what saves you from running out of space, slow I/O, or losing data. For the full filesystem layout and quota limits on each cluster, see Data transfer.

Where to put your files¶

The general rule: run everything from /scratch. Use /dipc for things you want to keep safe and small, and /lscratch when a single job needs very fast temporary I/O.

Checking your quota¶

getquota-home       # /dipc quota
getquota-scratch    # /scratch quota
getquota-data       # /data quota

getquota            # shows all quotas

When you run out of space¶

Running out of quota is the most common storage issue. Symptoms include failed jobs, "disk quota exceeded" errors, or inability to log in (if your home is full).

Quick fixes¶

1. Find what is using the space. From the filesystem that is full:

   du -sh /scratch/<username>/*  | sort -rh | head -20
   

2. Clean up common offenders:

   # Conda package cache (often several GB)
   conda clean --all
   
   # Apptainer image cache
   apptainer cache clean
   
   # Nextflow work directories from old runs
   nextflow clean -f
   
   # Pip cache
   pip cache purge
   

3. Find large old files. This only lists them; it does not delete anything:

   find /scratch/<username> -size +100M -mtime +30 -ls
   

   Review the output and manually remove what you no longer need.

Requesting a temporary quota increase¶

If you need more space temporarily (e.g., for a large dataset or a burst of simulations), contact us at support-hpc@dipc.org with:

• Which filesystem (/scratch, /data)
• How much additional space you need
• For how long
• A brief explanation of why

We can increase your quota temporarily. Once you are done, clean up and let us know so we can restore the default limits.

Small files and performance¶

Parallel filesystems like BeeGFS (Hyperion) are optimized for large sequential I/O, not for millions of small files. Conda environments, Python packages, and Node.js projects can generate tens of thousands of small files, which degrades performance for everyone on the shared filesystem and consumes your inode (file count) quota quickly.

If you have a large number of small files:

• Conda environments: package them inside an Apptainer container. The entire environment becomes a single .sif file.
• Datasets with many small files: combine them into a single archive (.tar) or use a format like HDF5.
• Temporary files: use /lscratch (local disk) instead of /scratch for jobs that create and delete many files during execution.

Sharing a directory with collaborators¶

Our clusters do not provide predefined group directories, but you can set one up yourself on any of the available filesystems. The recipes below work the same on /dipc, /scratch, or /data; pick the location that fits the lifetime of the data you want to share:

• /data (Hyperion only) for long-lived shared project data.
• /scratch for data being actively processed by jobs.
• /dipc for small shared scripts or configs (mind the home quota).

The examples assume the directory you want to share lives at /path/to/shared. Pick the approach below that matches your situation.

Group permissions¶

The simplest option when everyone who needs access already shares the same Unix group. The commands rely on standard chmod / rwx notation; if you already know it, jump straight to Setting up the share below. Otherwise, here is a quick recap.

Reading ls -l

   d   rwx   r-x   ---
   │    │     │     │
   │    │     │     └── others   no access
   │    │     └──────── group    read + traverse only
   │    └────────────── owner    read, write, traverse
   └─────────────────── type     d = directory · - = file · l = symlink

Three triplets in fixed order: owner → group → others. Each slot has slightly different meaning on files vs. directories:

              r              w               x
 file         read content   modify          run as a program
 directory    list (ls)      create/rename   enter (cd)

A - means the slot is not granted.

Changing with chmod

        chmod   u+x   script.sh
                │ │
       class ───┘ └── add  execute

 class       u owner    g group     o others    a all
 operator    + add      - remove    = set exact
 permission  r read     w write     x execute   X (dirs only)

chmod u+x   script.sh       # owner can execute it
chmod g-w   notes.txt       # remove write from the group
chmod a+r   notes.txt       # everyone can read
chmod -R g+rX project/      # recursive: group reads files, traverses subdirs

Setting up the share

getent group <group>

Set the group recursively on the directory:

chgrp -R <group> /path/to/shared

Grant read on files and read+execute on directories. The capital X is important: it adds execute permission only to directories (and to files that already had it), so plain data files do not become executable by accident:

chmod -R g+rX /path/to/shared

Every parent directory also needs +x for the group, otherwise members will not be able to cd into the shared directory:

chmod g+x /path/to

If group members should also be able to write to the directory, replace g+rX with g+rwX and set the setgid bit so new files inherit the group automatically:

chmod g+s /path/to/shared

Verify the result with:

ls -ld /path/to /path/to/shared
namei -l /path/to/shared/some_file

ACL-based permissions¶

Useful when collaborators do not all share a single Unix group, or when you prefer per-user access. Access Control Lists (ACLs) extend the basic owner/group/others model: instead of three permission triplets, you attach an arbitrary list of entries, each granting permissions to a specific user or group. The original owner/group/others bits stay in place; ACLs are layered on top.

A file or directory with ACLs shows a + at the end of its permission string in ls -l:

$ ls -l /path/to
drwxr-x---+ 2 <username> <group> 4096 May  5 10:23 shared

View the full list with getfacl:

$ getfacl /path/to/shared
# file: path/to/shared
# owner: <username>
# group: <group>
user::rwx
user:alice:r-x
group::r-x
group:project-x:r-x
mask::r-x
other::---

Each entry has the form <type>:<name>:<permissions>:

   u   :   alice   :   r-x
   │        │           │
   │        │           └── permissions (r/w/x bits, `-` for not granted)
   │        └────────────── name of the user or group (empty for the classic owner/group entries)
   └─────────────────────── type:  u = user, g = group, m = mask, o = other

The r/w/x bits mean exactly the same as in standard Unix mode:

              r              w               x
 file         read content   modify          run as a program
 directory    list (ls)      create/rename   enter (cd)

Entries without a name (e.g., user::rwx) correspond to the classic owner / file group / others bits and mirror what ls -l already shows. Named entries are the ACL additions: in the example above, user alice and group project-x are each granted r-x.

The setfacl flags used below are:

For an entire group:

# Applies to everything that already exists
setfacl -R -m g:<group>:rX /path/to/shared

# Default ACL: inherited by new files and directories created inside
setfacl -R -d -m g:<group>:rX /path/to/shared

# Parent directories need traversal permission for the group (r-x also works)
setfacl -m g:<group>:--x /path/to

For individual users (repeat per username):

setfacl -R -m u:<username>:rX /path/to/shared
setfacl -R -d -m u:<username>:rX /path/to/shared
setfacl -m u:<username>:--x /path/to

Inspect the configured ACLs at any time with:

getfacl /path/to/shared

To revoke access:

# Remove a specific user's entry, both from the current ACL and from the default ACL
setfacl -R -x u:<username> /path/to/shared
setfacl -R -d -x u:<username> /path/to/shared

# Remove all ACL entries, leaving only the standard owner/group/others bits
setfacl -R -b /path/to/shared

$ ls -ld /scratch/alice /scratch/alice/project-x
drwx------ 5 alice alice 4096 May  5 10:20 /scratch/alice
drwxr-x--- 3 alice alice 4096 May  5 10:23 /scratch/alice/project-x

setfacl -m u:bob:--x /scratch/alice
setfacl -m u:carol:--x /scratch/alice

setfacl -R -m u:bob:rX    /scratch/alice/project-x
setfacl -R -m u:carol:rwX /scratch/alice/project-x

setfacl -R -d -m u:bob:rX    /scratch/alice/project-x
setfacl -R -d -m u:carol:rwX /scratch/alice/project-x

$ getfacl /scratch/alice/project-x
# file: scratch/alice/project-x
# owner: alice
# group: alice
user::rwx
user:bob:r-x
user:carol:rwx
group::r-x
mask::rwx
other::---
default:user::rwx
default:user:bob:r-x
default:user:carol:rwx
default:group::r-x
default:mask::rwx
default:other::---

$ namei -l /scratch/alice/project-x/data.csv
f: /scratch/alice/project-x/data.csv
 drwxr-xr-x  root  root  /
 drwxr-xr-x  root  root  scratch
 drwx------+ alice alice alice
 drwxr-x---+ alice alice project-x
 -rw-r-----+ alice alice data.csv

$ getfacl /scratch/alice/project-x/results.dat
# file: scratch/alice/project-x/results.dat
# owner: carol
# group: carol
user::rw-
user:bob:r-x                    #effective:r--
user:carol:rwx                  #effective:rw-
group::r-x                      #effective:r--
mask::rw-
other::---

setfacl -R -x u:bob /scratch/alice/project-x
setfacl -R -d -x u:bob /scratch/alice/project-x
setfacl -x u:bob /scratch/alice