HPC & SLURM

HPCCompletionOptions

How the runner learns that a SLURM job has finished. Held on ModelManagerGlobals; adjust with setHPCCompletionOptions.

Jobs report their exit code by writing a sentinel file to a shared directory; the worker that submitted each job waits for its file. squeue is consulted only as a reaper, for jobs that died without writing anything, through one answer shared by every waiting worker.

Fields

• done_dir::String: Where sentinels are written. Empty means <dataDir()>/.hpc_done. Point it at a faster filesystem when the project lives on an NFS mount: NFS caches directory attributes for acdirmin/acdirmax (30s/60s by default), which delays how quickly a sentinel written on a compute node becomes visible here. Only this directory needs to move; data/ stays where it is.
• poll_interval::Float64: Seconds between each waiting worker's check for its own sentinel — one stat per in-flight job per interval. This is the completion path.
• reap_interval::Float64: How long one squeue answer is shared by every waiting worker before it is refreshed. This is only the reaper, so it is deliberately long; lowering it buys nothing for jobs that exit normally.
• grace_period::Float64: How long a job may be absent from the queue with no sentinel before it is declared failed. Covers the lag between a compute node writing the file and this node seeing it, so it must exceed the filesystem's worst-case directory-attribute staleness.

defaultJobOptions()

Return a Dict with default SLURM options.

Current defaults:

• "job-name" — simulation_id -> "S$(simulation_id)"
• "mem" — "1G"

isRunningOnHPC()

Return true if the current environment is an HPC environment, false otherwise.

Currently checks for a SLURM environment by probing for the sbatch command.

setHPCCompletionOptions(; kwargs...)

Set any of the HPCCompletionOptions fields on the active globals.

The default worth knowing about is done_dir: on a project whose data/ lives on an NFS mount, pointing it at a faster filesystem cuts completion latency without moving the project itself.

setHPCCompletionOptions(done_dir="/scratch/$(ENV["USER"])/mm_done")

setJobOptions(options::Dict)

Merge options into the global sbatch_options dictionary.

Each key–value pair becomes a --key=value flag appended to the sbatch command when running simulations on an HPC. Values that are Functions are called with the simulation ID at runtime.

shellCommandExists(cmd::Union{String,Cmd})

Check if a shell command exists in the current environment.

useHPC([use::Bool=true])

Set the global run_on_hpc flag to use.

Examples

useHPC()        # enable sbatch wrapping
useHPC(true)    # same
useHPC(false)   # run simulations locally