Running simulations

Once you have a trial, run executes it. The runner is generic: it prepares the trial, figures out which simulations are still pending, dispatches them in parallel, runs each one (locally or as a SLURM job), and writes results back to the database. The simulator backend only supplies the command via simulationCommand.

The two-phase run

run is split into two phases so that setup happens once and execution can be parallelized:

Preparation — the runner compiles shared code and materializes varied input folders for the whole trial. It calls the backend's setupSampling once per unique input-folder group and setupMonad for each monad.
Execution — the runner collects a SimulationSpec for every simulation that has not yet completed and launches each one inside its own task, asking the backend for the command via simulationCommand and handling the rest itself: output folder, output.log/output.err, working directory, and local-versus-SLURM dispatch.

Because preparation is separated from execution, the backend's simulationCommand can assume the monad is fully prepared and receives everything it needs in the SimulationSpec — no keyword arguments are threaded through.

output = run(inputs, dv; n_replicates=3)     # build + run in one call
# or, equivalently:
trial  = createTrial(inputs, dv; n_replicates=3)
output = run(trial)

run returns an MMOutput wrapping the trial that was executed, which you can pass straight back into createTrial/run to build follow-up trials from it.

Batching pre-built trials

If you accumulate trials in a vector, pass the whole vector to run (or createTrial) to launch them together as one parallelized batch:

sims = []
push!(sims, createTrial(inputs, dv1))
push!(sims, createTrial(inputs, dv2))
run(sims)                 # one Trial, one parallel pool across all constituent simulations

Elements may be any mix of Simulation, Monad, Sampling, or Trial (even in a loosely typed Vector{Any}); they are bundled into a single Trial, so run returns an MMOutput{Trial}. A non-trial element raises an ArgumentError.

Cheap re-runs

The runner only launches pending simulations. Because monads are keyed in the database by their parameterization (see The database), re-running a script reuses everything that already completed and runs only what is missing. To force fresh runs, set use_previous=false when constructing the trial.

Parallelism

By default ModelManager runs one simulation at a time. Raise the concurrency with setNumberOfParallelSims:

setNumberOfParallelSims(9)   # up to 9 simulations at once

Backends typically also honor an environment variable for this (for example PCMM reads PCMM_NUM_PARALLEL_SIMS) so the limit can be set without changing code:

PCMM_NUM_PARALLEL_SIMS=9 julia scripts/GenerateData.jl

The runner schedules simulations across Julia tasks up to this limit, regardless of which backend is in use.

Status tracking

Each simulation moves through the status codes from recognizedStatusCodes — Not Started → Queued/Running → Completed or Failed. The runner updates these as work progresses (updateDatabaseOnCompletion), so a query against the simulations table always reflects the current state of the campaign.

Simulations that end in Failed are removed from their monad's constituent list, so a failed parameter set leaves no record of the variation it came from. If failures are showing up during a calibration, When things go wrong covers how to read the per-generation failure files and what on_monad_failure does with them.

After each simulation

When a simulation completes, ModelManager runs the backend's postSimulationProcessing hook, then your optional post_processor callback, then the backend's postSimulationCleanup hook. The callback is how you compute and store quantities of interest from a finished run:

run(sampling; post_processor = QoI("counts", sp -> (; final_count = countCells(simulationID(sp)))))

See Post-processing and quantities of interest for the callback signature, what you may return, and how to read the results back.

For cluster execution, see HPC support. For the complete runner API (SimulationSpec, SimulationProcess), see the Runner reference.