Runner

SimulationProcess

Holds the outcome of a single simulation run.

Fields

• simulation::Simulation
• monad_id::Int
• process::Union{Nothing,Base.Process}: the local process, or nothing when the simulation ran as a SLURM job (there is no local process to hold) or no command could be built.
• success::Bool
• cmd::Union{Nothing,Cmd}: the command simulationCommand returned, or nothing if it could not build one.

process alone cannot tell a simulator hook what happened, because it is nothing for two unrelated reasons: a SLURM job (which ran, elsewhere) and a simulation that never had a command. cmd separates them — isnothing(cmd) means nothing was ever launched — and it is also the field to print when reporting a failure, since it is the simulator's own command on both paths rather than the sbatch wrapper.

SimulationSpec

A pending simulation to be launched. Produced by pendingSimulationSpecs and consumed by run, which wraps each spec in a @task that calls runSimulation on the active simulator.

monad_id is always a real monad ID — prepareTrialHierarchy always runs before spec collection, so setup is guaranteed to have completed.

Fields

• simulation::Simulation: The simulation to launch.
• monad_id::Int: ID of the enclosing monad. setupMonad has already run for this monad before the spec was built.

run(T::AbstractTrial; quiet=false, kwargs...) -> MMOutput

Run all pending simulations in T and return an MMOutput.

Keyword arguments

• quiet::Bool=false: when true, suppresses per-simulation and per-trial console output. Per-sim "Running simulation: N..." lines, the leading "Running ..." header, and the trailing "Finished ..." block are all gated by this flag. Used by ABC-SMC calibration to keep console output focused on per-generation progress.

• on_progress::Union{Nothing,Function}=nothing: optional progress hook. When supplied, it is called as on_progress(:init, n_simulation_tasks) once after the pending simulation count is known, on_progress(:step, 1) after each simulation completes, and on_progress(:finish, n_success) once at the end. When nothing (default) the runner behaves exactly as before — this keeps the per-simulation completion loop framework- agnostic while letting callers (e.g. ABC-SMC calibration) render a live progress bar.

• post_processor::Union{Nothing,Function}=nothing: optional user hook run once per successfully completed simulation, after the simulator's non-destructive postSimulationProcessing and before its destructive postSimulationCleanup — so the callback always sees the intact (but processed) output folder. It is called as post_processor(simulation::Simulation) — the same argument a QoI's compute receives, so one measurement function serves the sink, sensitivity analysis and calibration alike. A QoI or a vector of them may be passed instead of a function. Use simulationID and pathToOutputFolder(simulation) rather than reaching into fields; the hook only fires for simulations that succeeded, and the owning monad is only(monadIDs(simulation)) (which queries the database, and throws if the simulation is gone); reading the actual simulation output into usable data is the responsibility of the user or the simulator package (e.g. PhysiCellModelManager loaders keyed by simulationID). Its return value determines storage:

  • nothing → nothing is stored (pure side effects).
  • a NamedTuple or AbstractDict of name => scalar → one row keyed by simulation_id is upserted into the project's post-processing sink (data/outputs/postprocessing.db), readable via postProcessingTable. Each key becomes the column "<qoi name>.<key>", so two measurements that both report a tumor stay separate; a scalar return uses the name alone. Columns grow dynamically; sims lacking a given quantity have NULL.
  • any other type → an ArgumentError is thrown.

  Because every column is named after the QoI that wrote it, a bare anonymous function that stores anything is refused: its derived name is a gensym that changes between sessions, so the same script would write a fresh, half-empty set of columns each run. Wrap it — QoI("counts", sim -> …) — or pass a named function. A callback returning nothing is unaffected. The callback runs inside the per-simulation worker task (so heavy compute parallelizes), but all sink writes are serialized in the main completion loop; user code never touches the sink DB directly. post_processor is not forwarded to the simulator hooks. If the callback (or a simulator hook) throws, run fails fast: it rethrows a clear error naming the stage and simulation with the original stacktrace — it never hangs or swallows the exception.

• All other kwargs flow through to prepareTrialHierarchy (which forwards them to the simulator's setupSampling / setupMonad hooks) and to both postSimulationProcessing and postSimulationCleanup (e.g. prune_options). Any simulator-specific flags flow through this channel. runSimulation takes no kwargs.

• run_kwargs::NamedTuple=(;): simulator options as a bundle, equivalent to passing them loosely. Calibration must bundle — runABC and resumeCalibration spend their keyword splat on ABCSMC fields — so accepting the bundle here means one assembled once is portable to any entry point. Where a key appears both ways, the loose keyword wins.

monadID(simulation_process::SimulationProcess)

Return the ID of the monad enclosing this simulation.

pathToOutputFolder(simulation_process::SimulationProcess)

Return the path to the output folder for the simulation this process ran.

prepareTrialHierarchy(T::AbstractTrial; kwargs...) → Bool

Recurse down the trial hierarchy, creating output folders and calling the simulator's setupSampling and setupMonad hooks. Returns true on success, false if any hook fails (in which case the remaining hierarchy is skipped).

kwargs are forwarded to both hooks — any simulator-specific flags flow through this channel. This function has no knowledge of console output and does not touch simulation status codes.

Dispatch behaviour:

• AbstractMonad (Simulation or Monad): mkpath + setupSampling on M (compile code, etc.) + setupMonad on M (prepare varied input folders).
• Sampling: mkpath + setupSampling once for the whole sampling + mkpath and setupMonad for each constituent monad. setupSampling is called only once, not once-per-monad.
• Trial: mkpath + recurse into each sampling.

runSimulation(sim::AbstractSimulator, spec::SimulationSpec) → SimulationProcess

Run the simulation described by spec and report how it went. This default asks the backend for the command via simulationCommand and does everything else: it creates the simulation's output folder, sends stdout and stderr to output.log and output.err there, runs the command in simulatorDir (or the Cmd's own dir), and – when run_on_hpc is set – submits it as a SLURM job instead and waits for it. Called by run inside each worker task.

A backend only overrides this if its simulation is not an external process; for those, SimulationProcess.process may be nothing.

simulationCommand may return nothing to say no command could be built for this simulation; that is recorded as a failed simulation and the rest of the trial continues.

Otherwise the command must be a bare Cmd. Redirections and pipelines are added here, so a pipeline(...) is rejected; and it must not carry an environment (Cmd(...; env=...), setenv, addenv). Julia's Cmd.env replaces the environment, so locally the simulation would see only the variables listed; as a SLURM job the environment is not forwarded at all and the job inherits the submitting one. The same command would mean two different things, so it is refused rather than made silently path-dependent. Put what the simulation needs in the command's arguments or its working directory.

A local process that fails to start (missing executable, unwritable folder) is recorded as a failed simulation, not raised: one broken simulation should not abort a campaign of thousands. A process killed by a signal is also a failure – Julia reports exitcode == 0 for those, so the check is success(p), not the exit code.

simulationID(simulation::Simulation)
simulationID(simulation_process::SimulationProcess)

Return a simulation's ID.

A post_processor (see run) is called with a Simulation, so that is the form to use in one — simulationID(sim) rather than reaching for sim.id. The SimulationProcess method serves the runner and the AbstractSimulator hooks (postSimulationProcessing, postSimulationCleanup), which still receive that type.

wasSuccessful(simulation_process::SimulationProcess)

Return true if the simulation completed successfully.

QoI(name, compute; reduce=mean)

A named quantity of interest: measure something per simulation, then combine the replicates.

Three parts of ModelManager need a number out of a group of simulations — sensitivity analysis, calibration, and the post-processing sink — and each used to ask in its own shape. A QoI is that measurement written once and passed to any of them.

Arguments

• name: identifies the quantity. It is the sink's column name and the key under which CalibrationProblem reports the value to its distance. It may not contain a ., which is reserved as the separator between a quantity and its components (see below).
• compute: called with one Simulation. It may return anything reduce understands — a scalar, a vector, a Dict — except when the QoI is used as a post_processor, where reduce is never called and compute's own return value is what gets stored.

Keywords

• reduce: collapses one parameter set's replicates, mean by default. It receives the vector of everything compute returned for that set — one entry per replicate — and returns that set's value. It is not Base.reduce, and the result need not be a scalar: what it reduces is the replicate dimension, so returning a Dict of name => value is a reduction over replicates that keeps several quantities, and every consumer here understands one.

What each consumer needs back

Neither compute nor reduce is constrained by QoI itself; the requirement comes from where the QoI is used, and it does not fall on the same function in each case:

The sink is the exception, and the reason is that it fires once per simulation: there is exactly one value and nothing to combine, so reduce is never called and the freedom to return a vector does not apply. Write richer per-simulation output to the simulation's own folder instead. Returning something a consumer cannot use is that consumer's error to raise, and the sink's names the QoI and the offending type.

A Dict becomes several quantities, not one

Two consumers spread a keyed value rather than demanding one number, and both name the pieces the same way — "<qoi name>.<key>" — so one measurement names its parts identically wherever it is used:

• Sensitivity analysis runs one analysis per key, labelled "<qoi name>.<key>" — so QoI("counts", …) reducing to Dict("tumor" => …, "immune" => …) gives counts.tumor and counts.immune. Every monad must reduce to the same keys; one that does not is refused, because a sensitivity index computed over a missing value is wrong rather than approximate. A Vector is not spread by index: only its length could be checked against the other monads', and equal length is not equal meaning.
• The sink names its columns the same way: "<qoi name>.<key>". Because those names are persisted, an anonymous compute is refused outright when it spreads — its derived anon_9 would prefix every column and vary between sessions. Name the QoI, or pass a named function.

Because reduce sees every replicate's value, a measurement that needs the replicates jointly rather than as summarised numbers is expressed by having compute return the raw material — a time series, say — and letting reduce do the pooled work.

reduce is the monad-level step, not merely an average

This matters whenever the quantity involves a nonlinearity applied after the replicates are combined. Sensitivity analysis on a discrepancy-to-data score is the common case: you want each measured value averaged across replicates and then compared to data, because averaging squared errors is not the same number as squaring the averaged error. A per-simulation compute cannot do it — it has no access to the mean — but reduce can, because it receives every replicate:

observed = Dict("tumor" => 2.0, "immune" => 3.0)

QoI("mse", endpointCounts;
    reduce = per_sim -> sum((mean(getindex.(per_sim, k)) - observed[k])^2 for k in keys(observed)))

compute returns each simulation's raw values, and reduce averages per quantity, takes the squared differences, and sums them into the one number sensitivity analysis needs. Reporting spread alongside the mean is a second QoI over the same compute, with reduce comparing std to the observed spread instead.

Reading a value the sink stored earlier

Post-processing runs while a simulation's output folder still exists; post-simulation cleanup may then delete what the quantity was computed from. A QoI whose compute reads the sink instead of the output folder therefore still works afterwards, and needs nothing new — the sink is keyed by simulation ID and by the QoI's own name:

tumor = QoI("tumor", s -> finalPopulationCount(s)["tumor"])
run(trial; post_processor=tumor)                      # stores it while the output exists

stored = QoI("tumor", s -> postProcessingTable([s.id]).tumor[1])
run(MOAT(), spec; functions=[stored])                 # reads it back, output folder or not

stored automates the lookup, and defaults to :never:

QoI("tumor", computeFromOutput; stored=:prefer)    # stored value if present, else compute
QoI("tumor", computeFromOutput; stored=:require)   # stored value or an error

It defaults off because nothing records which compute produced a stored value, and no fingerprint can: redefining a function's body in place leaves both hash and nameof unchanged, so a changed compute is undetectable, while two textually identical anonymous functions hash differently, so an unchanged one is equally unrecognisable. The sink stores no provenance either.

What is robust is recomputation. verifyStoredValues recomputes where a simulation's output survives and reports agreements, mismatches, and how many could not be checked because the output is gone — which is precisely the case stored exists for. Run it before trusting stored values for anything that matters.

Examples

tumor = QoI("tumor", s -> finalPopulationCount(s)["tumor"])

# A different way of combining replicates
QoI("tumor_median", s -> finalPopulationCount(s)["tumor"]; reduce=median)

# Spread across replicates, rather than their centre
QoI("spread", s -> finalPopulationCount(s)["tumor"]; reduce=std)

# Pooling the replicates instead of reducing per-simulation numbers
QoI("slope", timeSeries; reduce=series -> fitSlope(reduce(vcat, series)))

# One QoI, three consumers
run(MOAT(), spec; functions=[tumor])
CalibrationProblem(spec, observed, tumor, mseDistance)
run(trial; post_processor=tumor)

# Several quantities from one measurement, named the same way in both places: GSA spreads
# `reduce`'s keys into `counts.tumor` / `counts.immune`, and the sink spreads `compute`'s keys
# into columns `counts.tumor` / `counts.immune`.
counts = QoI("counts", finalPopulationCount;
             reduce = per_sim -> Dict(k => mean(getindex.(per_sim, k)) for k in ("tumor", "immune")))

verifyStoredValues(q::QoI, T; rtol=1e-8, limit=nothing) → NamedTuple

Check a QoI's stored values against freshly computed ones, for the simulations of T.

Returns (; n_checked, n_agreed, n_mismatched, n_unverifiable, n_missing, mismatches). A simulation is unverifiable when its output folder is gone, which is exactly the situation stored exists for — the value may be perfectly good, but nothing here can confirm it.

Use this before trusting stored=:prefer or stored=:require on results you care about. Nothing about a stored value records which compute produced it, so recomputation is the only real check.

Example

report = verifyStoredValues(tumor, my_sampling)
# `n_mismatched == 0` alone is NOT a pass: it is also what you get when every simulation was
# skipped. Require that something was actually compared.
report.n_agreed > 0 || error("nothing was verified: $(report.n_missing) had no stored value " *
                             "and $(report.n_unverifiable) had no output folder to recompute from")
report.n_mismatched == 0 || error("stored values disagree with a fresh computation")

StudySpec(inputs::InputFolders, variations; kwargs...)
StudySpec(reference::AbstractMonad, variations; kwargs...)

The model-and-parameters half of a study, built once and used for either sensitivity analysis or calibration.

A sensitivity sweep and a calibration ask different questions of the same model varied over the same parameters. StudySpec is that shared half — the input folders, the parameters, the baseline to vary from, and how many replicates to run — so it need not be restated when the second question follows the first.

Arguments

• inputs: the model's input folders. The reference form takes them from a monad instead, along with its variation ID as the baseline.
• variations: a vector of AbstractVariations, or several passed individually.

Keywords

• reference_variation_id: the baseline to vary from. Defaults to VariationID(inputs); the reference form takes it from the monad and does not accept this keyword.
• n_replicates: replicates per parameter set (default 1).
• use_previous: reuse matching simulations that have already run (default true). Sensitivity only — calibration reuses through its own SimulationBank, so this field is ignored there.

What it deliberately does not hold

observed_data, summary_statistic and distance stay on CalibrationProblem, and functions stays on the sensitivity entry point. A sensitivity study has no observed data, and a field that half the consumers ignore is how a shared abstraction rots.

The user's own variations are kept rather than normalised, because the reverse conversion is lossy: a DistributedVariation's display name does not survive it, and the generation CSVs are keyed by that name.

Examples

spec = StudySpec(inputs, [dv1, dv2]; n_replicates=3)

# Sensitivity, then calibration, over the same spec
gsa  = run(MOAT(), spec; functions=[finalCount])
prob = CalibrationProblem(spec, observed, summarize, mseDistance)
res  = run(ABCSMC(population_size=64), prob)

# From a monad, which supplies both the inputs and the baseline variation
spec2 = StudySpec(reference_monad, [dv1, dv2])