Job priority and scheduling¶

When you submit a job, it does not necessarily run straight away: it joins a queue. The order in which queued jobs are dispatched is determined by a priority value that SLURM recalculates periodically for every pending job.

This page explains how that priority is computed, why your job may wait longer than another user's even when you have asked for fewer resources, and how to inspect the relevant numbers yourself.

The four priority factors¶

The priority of a pending job is the weighted sum of four contributions:

• QoS: a contribution determined by the QoS you select (regular, xlong, test, ...). Each QoS has its own configured priority value, so different QoS choices produce different contributions to a job's total priority. See Partitions and QoS for the available QoS levels.
• Fairshare: a measure of how much you have been consuming recently, relative to your assigned share of the cluster. Heavy recent usage drives this value down. A quiet period drives it back up.
• TRES (Trackable RESources): a small contribution proportional to the resources requested (CPU, memory, GPUs).
• Age: a contribution that grows the longer the job sits in queue. It is calibrated to become a meaningful counterweight for long waits, see The age contribution below.

The full formula¶

SLURM computes job priority as a weighted sum of several factor contributions. The general form is:

priority = weight_age       × age_factor       +
           weight_assoc     × assoc_factor     +
           weight_fairshare × fairshare_factor +
           weight_jobsize   × jobsize_factor   +
           weight_partition × partition_factor +
           weight_qos       × qos_factor       +
           SUM(weight_tres × tres_factor) over each tracked resource
           + site_adjustment - nice_value

Each *_factor is a normalised value between 0 and 1, and each weight_* is its configured maximum. On our clusters only the age, fairshare, QoS and per-TRES weights are set to non-zero values. The others contribute nothing. The site adjustment and nice value play no role in normal job submission either. This is why we describe the priority calculation in terms of just four factors above: those are the only ones that actually move.

Weights and actual contributions¶

These factors do not contribute equally. The weights configured on our clusters are:

Each weight is the maximum contribution that factor can make to a job's total priority. The value you actually see in sprio for a given job is the weight multiplied by a normalised factor between 0 and 1: a QoS contributes proportionally to its own configured priority value (relative to the highest-priority QoS on the cluster), fairshare contributes proportionally to your current effective share, age grows linearly with wait time up to a cap of 30 days on our clusters, and TRES scales with the resources requested.

A practical consequence is that two jobs submitted to the same partition can show very different QoS contributions in sprio depending on which QoS each one used. You may see QoS contributions of 10.000.000, 50.000.000 or 100.000.000 sitting next to each other in the same sprio output. They reflect different QoS choices, not different "weights" in any per-user sense.

The hierarchy of weights is deliberate. QoS sets a coarse priority class. Within a given QoS, fairshare is the dominant driver of who runs next, while age grows into a meaningful counterweight for jobs that wait long enough (see The age contribution). TRES acts as a tie-breaker.

QoS priority tiers¶

Every QoS configured on our clusters has a Priority value, and the QOS_factor for a job is that priority divided by the highest priority among all QoS (currently 2000). The contribution that appears in the QoS column of sprio is therefore:

contribution = weight_qos  × Priority(this QoS) / max(Priority across all QoS)
             = 100.000.000 × Priority(this QoS) / 2000

The QoS levels currently defined on our clusters fall into three priority tiers. The exact list of QoS in each tier may differ slightly between clusters, but the tiering structure is consistent:

Access to a given QoS depends on your account and group memberships. Group-exclusive QoS in the priority-2000 tier are restricted to members of the corresponding research groups. For most users, production work runs under one of the priority-200 QoS (regular, long, xlong, ...) and test is intended for short interactive runs. See Partitions and QoS for which QoS you can submit jobs to.

You can list the current values yourself with:

sacctmgr show qos format=name,priority

Fairshare under Fairtree¶

Our clusters compute fairshare with the Fairtree algorithm. The accounting tree is intentionally flat: every user sits directly under the root association, and every user is assigned the same share of the cluster. There are no intermediate group accounts and no per-group entitlements; entitlement is per user, and identical for everyone.

Within that flat structure the algorithm is conceptually simple:

• Fairtree compares each user's recent usage against their (equal) assigned share.
• Each user is given a normalised fairshare score in [0, 1] reflecting their position in that comparison relative to all other users.
• A user who has never run jobs has fairshare = 1.0, the perfect score: their assigned share is fully unused.
• A user who has consumed heavily relative to their entitlement has fairshare close to 0.
• The contribution to job priority is the fairshare weight (10.000.000 maximum) multiplied by fairshare_factor, so a pristine user gets the full 10.000.000 and a heavily-used user gets close to 0.

In practice the cluster always carries a mix of pristine users (fairshare = 1.0) and active users whose fairshare has decayed below 1.0. The mean fairshare among active users (those with fairshare < 1) is the natural reference point when we talk about a "typical" fairshare contribution: it reflects what a user who has been working steadily on the cluster looks like in sprio right now. This reference value is also what we use to calibrate the age contribution, described next.

You can inspect your own values with:

sshare -u $USER          # your fairshare and recent usage
sshare -a                # the full association tree

The age contribution¶

The age weight on our clusters is 2.000.000 points, accumulated linearly over 30 days. This value is not arbitrary: it is calibrated so that a job which has waited the maximum age receives a contribution comparable to the mean fairshare contribution among active users on the cluster.

The reasoning is straightforward. Without this calibration, a heavy user with a very low fairshare could sit behind pristine users indefinitely. With it, the longer a job waits, the more its age contribution grows toward the typical fairshare contribution on the cluster, so no job is permanently outranked by fairshare alone: long-waiting jobs catch up over time.

Concretely, with the current configuration:

• A pristine user (fairshare = 1.0) contributes 10.000.000 to the fairshare column.
• A typical active user contributes around 2.000.000 to the fairshare column, depending on the current state of the cluster.
• A job waiting the full 30 days contributes 2.000.000 to the age column, matching that typical active user.

The exact crossover point depends on the current mean fairshare among active users, which fluctuates as people submit and complete work. The intent is calibration to a moving target, not an exact equality at any particular instant.

A worked example¶

Consider two pending jobs from different users in the general partition. User A's job has been waiting 5 days and user B's job 8 days:

  JOBID  PARTITION  PRIORITY  SITE     AGE  FAIRSHARE       QOS  TRES
5539493  general    11759742     0  333333    1424261  10000000  cpu=23,mem=70,gres/gpu=...   (user A)
5534600  general    10945779     0  533333     410380  10000000  cpu=6,mem=6,gres/gpu=...     (user B)

Both users picked the same QoS, so the QoS contribution is identical (10.000.000, which is what the chosen QoS contributes, not the maximum a QoS can contribute on this cluster). With the current age weight, the AGE column carries real signal: 533.333 for user B and 333.333 for user A, which is wait_days / 30 × 2.000.000 in each case. The TRES contributions are similar and small for both jobs.

The decisive factor is still fairshare: 1.424.261 vs 410.380, a gap of roughly one million points. Age narrows that gap in user B's favour by around 200.000 points, which is no longer trivial, but the fairshare difference is roughly five times larger and accounts for most of the remaining ~800.000-point gap in total priority. User A will be scheduled first.

The mechanism is doing exactly what is intended: as user B's job keeps waiting, its AGE contribution keeps growing toward the 30-day cap of 2.000.000, which would more than overturn a fairshare gap of this size. Long-waiting jobs catch up over time.

This is not a configuration problem or a per-user privilege. Both users have the same entitlement to the cluster. User A simply has more fairshare available right now because they have consumed fewer resources in the recent past.

Why fairshare dominates in practice¶

Within a given QoS, fairshare is the only factor with both a large weight and a wide dynamic range, which makes it the practical determinant of who runs next. The mechanism is straightforward:

• When you run jobs, your accumulated usage grows and your fairshare value decreases.
• When you do not run jobs, that accumulated usage decays continuously and exponentially. On our clusters the half-life is 7 days: every seven days of light use roughly halves your accumulated usage.
• After one week of light use you have recovered about halfway back to your baseline. Reaching most of the way back takes two to three weeks.

The practical implication: if you have just finished a heavy campaign and your next batch of jobs is suddenly waiting longer than a colleague's, that is the system doing exactly what it is designed to do, distributing access fairly over time rather than at every instant.

Backfilling¶

Priority is the primary mechanism but not the only one. SLURM also runs a backfill scheduler, which lets lower-priority jobs start ahead of higher-priority ones, provided doing so does not delay the projected start time of any higher-priority job.

Backfilling is why a small, short job from a user with low fairshare can sometimes start before a large, long job sitting at the head of the queue. The large job is being held for a future window of resources. In the meantime, smaller jobs that fit into the gap are allowed through.

Inspecting priority and fairshare yourself¶

Two commands are useful whenever you want to understand why a job is still pending:

# Priority breakdown of a specific job
sprio -j <JOBID>

# Your current fairshare and recent usage
sshare -u $USER

sprio shows exactly the column breakdown used in the worked example above: how each factor is contributing to the priority of your job at this moment. sshare shows your raw and effective fairshare values together with your recent usage, which is what feeds the fairshare column in sprio.

If after looking at these numbers something still does not add up, for example your fairshare seems implausibly low given your actual usage, or a job is in a state you do not expect, please open a ticket with support-hpc@dipc.org and include the relevant sprio and sshare output.