LoginSign Up
← Back to all sparks
S

Slurm

DEVOPS
Velocity5.0

Workload manager and job scheduler for HPC clusters

slurm.schedmd.com ↗

Slurm 26.05 makes SLUID the default job identity and pulls TLS into the daemons.

hpcjob-schedulingreliabilitytlsupgrade-compatibilityaccounting
◆Current state
Slurm is maintaining three branches concurrently — 26.05 as the new major line, 25.11 as the stable one, and 25.05 still receiving fixes — and releases them in same-minute pairs. The 26.05 line introduces SLUID as the default identifier used in cgroup paths and step lookups, and a tls/s2n path with a certgen plugin for generating step certificates. Everything else in this window is correctness work: controller crashes under load, shutdown and reconfigure races, memory leaks in slurmstepd and slurmrestd, and cross-version compatibility between mismatched daemons.
◆Where it's heading
The bulk of the engineering effort visible here goes to concurrency and lifecycle correctness in slurmctld — races between the purge thread and job/node write locks, accounting_storage callers racing plugin teardown, connection plugin initialization ordering, forwarding threads outliving shutdown. That is the profile of a codebase being hardened for larger and busier clusters rather than one adding scheduling features. The second thread is upgrade tolerance: multiple fixes exist purely to keep a 26.05 slurmd working with a 24.11 sattach or a 25.05 slurmstepd, which tells you how staged real site upgrades are.
◆Prediction
Expect 26.05.x to keep issuing fixes for the SLUID transition and for mixed-version daemon combinations, since both are where this window's regressions cluster. The 25.05 line looks close to the end of its fix stream, with its recent releases limited to shutdown races and build compatibility.

◆Recent moves

  1. 25d ago

    26.05.2 adds a cgroup path escape hatch back to job IDs

    Restores the option to use the job ID rather than SLUID in cgroup paths, with SLUID still the default — an acknowledgement that the 26.05 identity change broke site tooling. Also fixes stale QOS and association pointer dereferences in slurmctld and an X11-forwarding deadlock in the s2n TLS path.

    View source ↗
  2. 25d ago

    25.11.7 fixes requeued jobs losing their license reservations

    The notable fix here is a 25.05 regression where requeued jobs lost their license and HRES requests, letting Slurm run them without sufficient licenses — a correctness bug with direct licensing-compliance consequences. The rest is controller crash and lock-contention work shared with the 26.05 line.

    View source ↗
  3. 2mo ago

    26.05.1 repairs mixed-version and post-upgrade job handling

    Concentrated on the upgrade path: step start and completion for jobs that survived a 25.05 to 26.05 upgrade, a 26.05 slurmd hanging a 24.11 sattach, blocked memory resize for pre-26.05 jobs, and DB state replay leaking on version mismatch. A first point release that is almost entirely about the transition it just created.

    View source ↗
  4. 2mo ago

    Slurm 26.05 ships SLUID identity and an in-tree TLS path

    ⚡ SPARK

    The major release that sets up everything the point releases have been repairing since: SLUID becomes the default identity used for cgroup paths and step lookup, TLS lands through an s2n plugin with certificate generation, and slurmdbd storage can now be reached over a Unix socket. The fix-heavy releases that follow are the cost of those two changes.

    View source ↗
  5. 2mo ago

    25.11.6 clears reconfigure and shutdown hangs in slurmctld

    A dense correctness release: reconfigure and shutdown hangs caused by unenforced client I/O timeouts, a startup crash when I/O beat the connection plugin's initialization, deadlock when replacing nodes in reservations, and slow scheduling for multi-segment topology/block jobs. Representative of where this project's effort actually goes between majors.

    View source ↗
  6. 2mo ago

    25.05.8 rejects nobody-user submissions and fixes shutdown races

    Rejects job submissions running as the reserved nobody(99) user or group and blocks --uid=99 and --gid=99 outright, plus a sweep of tree-forwarding shutdown races across every daemon. The uid 99 rejection is a behaviour change, not just a fix, and sites relying on that identity will notice.

    View source ↗