Database

getParameterValue(M::AbstractMonad, xp::XMLPath)
getParameterValue(M::AbstractMonad, xml_path::AbstractVector{<:AbstractString})
getParameterValue(simulation_id::Int, xp)

Get the parameter value for the given XML path from the monad's variations database if the column exists, otherwise fall back to the base XML file.

The location is inferred from the XMLPath. For the generic 3-argument form (explicit location), see ModelManager.getParameterValue.

• Boolean strings ("true" / "false") are returned as Bool.
• Numeric strings are returned as Float64.
• Everything else is returned as-is.

buildWhereClause(table_name::String, ids::Vector{<:Integer}, filters::Dict; db::SQLite.DB=centralDB())

Build a WHERE clause with ids and optional column filters.

calibrationsSchema()

Return the SQL schema string for the calibrations table.

provenance_id references the provenances row recording the session, script, and git state that created the run, exactly as on the trial tables. An existing database gains the column from ensureProvenanceColumns on its next initializeModelManager, with no migration.

constructSelectQuery(table_name::String, condition_stmt::String=""; selection::String="*")

Build a SELECT SQL string for table_name.

createMMTable(table_name::String, schema::String; db::SQLite.DB=centralDB())

Create a table in the central database if it does not already exist.

The table name must end in "s" (for ID naming conventions) and the schema must include a PRIMARY KEY column named <table_singular>_id.

databaseDiagnostics(max_ids::Dict{Type{<:AbstractTrial},Int} = Dict{Type{<:AbstractTrial},Int}())

Check consistency between the database and the output folders. Prints warnings for any discrepancies found.

When max_ids is provided (as returned by _snapshotMaxIDs), each check is restricted to IDs ≤ the snapshot value for that type. This prevents false positives from simulations that were created or started after initializeModelManager returned.

initializeDatabase()

Initialize the central database, creating the schema if it does not already exist.

insertFolder(location::Symbol, folder::String, description::String="")

Insert folder into the database for location (if not already present), create its per-folder variations SQLite database, and call initializeInputFolder on the active simulator.

locationVariationsDatabase(location::Symbol, folder::String)

Return a SQLite.DB connection to the per-folder variations database for location/folder, nothing if the folder is empty (location unused), or missing if the variations DB file does not exist.

monadsTable(args...; kwargs...)

Return a DataFrame with one row per monad and its varied parameters — the monad-level analogue of simulationsTable. See monadsTableFromQuery for keyword arguments.

args... can be:

• Any AbstractTrial objects (or arrays thereof) — the monads they contain are collected via monadIDs.
• A vector of monad IDs.
• Omitted (returns data for all monads).

A Simulation contributes the monad holding it, so a bare simulation gives a one-row table. In the rare case that a simulation has no monad — see monadIDs — the result is an empty table rather than an error.

Examples

monadsTable(sampling)

monad_ids = [1, 2, 3]
monadsTable(monad_ids; remove_constants=false)

monadsTable(sampling; tags=true)   # adds a tag:<key> column per tag key in use

monadsTableFromQuery(query::String; remove_constants::Bool=true, sort_by=String[], sort_ignore=String[], short_names::Bool=true, tags::Bool=false, include_auto_tags::Bool=false)

Return a DataFrame for the given SQL query on the monads table. This is the monad-level analogue of simulationsTableFromQuery: one row per monad and its varied parameters.

Keyword arguments match simulationsTableFromQuery, with two differences: the display ID column excluded from the default sort is :MonadID (rather than :SimID), and there is no post_processing option, since the sink is keyed by simulation. tags=true appends tag:<key> columns as it does for simulations, inheriting from a parent Sampling or Trial.

postProcessingDBPath()

Return the path to the project's post-processing sink database (<data_dir>/outputs/postprocessing.db). The file is created lazily the first time a post_processor (see run) returns quantities of interest to store.

postProcessingTable(args...)

Return a DataFrame of stored post-processing quantities of interest, one row per simulation (keyed by :SimID). See run's post_processor keyword for how rows are produced. The result is joinable to simulationsTable on :SimID.

args... can be:

• Any AbstractTrial objects (or arrays thereof) — their simulations are collected via simulationIDs.
• A vector of simulation IDs.
• Omitted (returns data for all simulations that have stored quantities).

Simulations without stored quantities are absent from the table; quantities not computed for a given simulation appear as missing. Returns an empty table if no post-processing has run.

Examples

out = run(sampling; post_processor = QoI("cells", sp -> (; final_count = countCells(simulationID(sp)))))
postProcessingTable(sampling)      # column: cells.final_count

printMonadsTable(args...; sink=println, kwargs...)

Print a table of monads and their varied values. See monadsTable.

Keyword Arguments

• sink: A function to receive the DataFrame (default println). Can also use CSV.write.

Examples

printMonadsTable([monad_3, sampling_2, trial_1])

using CSV
printMonadsTable(; sink=CSV.write("temp.csv"))

printPostProcessingTable(args...; sink=println)

Print the post-processing quantities-of-interest table. See postProcessingTable.

Keyword Arguments

• sink: A function to receive the DataFrame (default println). Can also use CSV.write.

printSimulationsTable(args...; sink=println, kwargs...)

Print a table of simulations and their varied values. See simulationsTable.

Keyword Arguments

• sink: A function to receive the DataFrame (default println). Can also use CSV.write.

Examples

printSimulationsTable([simulation_1, monad_3, sampling_2, trial_1])

sim_ids = [1, 2, 3]
printSimulationsTable(sim_ids; remove_constants=false)

using CSV
printSimulationsTable(; sink=CSV.write("temp.csv"))

queryToDataFrame(query::String; db::SQLite.DB=centralDB(), is_row::Bool=false)

Execute query and return the result as a DataFrame.

If is_row is true, asserts that exactly one row is returned.

recognizedStatusCodes()

Return the list of valid simulation status code strings.

reinitializeDatabase()

Reinitialize the database, scanning data/inputs/ to register any new folders.

shortLocationVariationID(fieldname::Symbol)
shortLocationVariationID(fieldname::String)
shortLocationVariationID(type::Type, fieldname)

Return the abbreviated column-name symbol for fieldname's variation ID in display tables.

Dispatches to shortLocationVariationID(simulator(), fieldname). The default implementation returns locationVariationIDName(fieldname) |> Symbol. Simulator packages should extend shortLocationVariationID(::TheirSimulator, fieldname::Symbol) to provide custom abbreviations.

shortVariationName(location::Symbol, name::String)

Return the display name for variation column name at location.

Dispatches to shortVariationName(simulator(), location, name). The default returns name unchanged. Simulator packages should extend shortVariationName(::TheirSimulator, location, name) to provide human-readable column names.

simulationsTable(args...; kwargs...)

Return a DataFrame with simulation data. See simulationsTableFromQuery for keyword arguments.

args... can be:

• Any AbstractTrial objects (or arrays thereof)
• A vector of simulation IDs
• Omitted (returns data for all simulations)

Pass post_processing=true to append each simulation's stored post-processing quantities (see postProcessingTable) as extra columns, and tags=true to append a tag:<key> column per tag key in use (see appendTags!):

simulationsTable(sampling; post_processing=true)
simulationsTable(sampling; tags=true)                        # what each run was for
simulationsTable(sampling; tags=true, post_processing=true)  # intent and outcome together

The two answer different questions and are often read side by side: tags record what a run was for, the post-processing sink records how it turned out.

simulationsTableFromQuery(query::String; remove_constants::Bool=true, sort_by=String[], sort_ignore=String[], short_names::Bool=true, post_processing::Bool=false, tags::Bool=false, include_auto_tags::Bool=false)

Return a DataFrame for the given SQL query on the simulations table.

By default, constant columns and raw ID columns are removed.

Arguments

• query::String: The SQL query to execute.

Keyword Arguments

• remove_constants::Bool: If true, removes columns that have the same value for all simulations. Defaults to true.
• sort_by::Vector{String}: A vector of column names to sort the table by. When empty (the default), sorts by every parameter column in table order (i.e. the first parameter column is the primary key), excluding :SimID and the variation-ID columns. :SimID is not sorted by default but may be requested explicitly here. To populate this argument, it is recommended to first print the table to see the column names.
• sort_ignore::Vector{String}: Additional column names to exclude from sorting, on top of the always-excluded variation-ID columns. Defaults to none.
• short_names::Bool: If true (default), column names are shortened via shortVariationName. Pass false to keep raw XML-path column names (e.g. for matching against parameters.toml db_column entries).
• post_processing::Bool: If true, left-joins each simulation's stored post-processing quantities (see postProcessingTable) onto the table by :SimID, appending one column per quantity (missing where a quantity was not computed). Defaults to false. Post-processing columns are appended as-is and are not subject to remove_constants or sorting.
• tags::Bool: If true, appends one tag:<key> column per tag key in use (see appendTags!), with missing where a simulation has no value for that key and multiple values joined by |. Defaults to false. A tag on a parent Monad, Sampling, or Trial contributes to its simulations' columns, matching findSimulationIDs. Like post-processing columns, these are appended as-is and are not subject to remove_constants or sorting.
• include_auto_tags::Bool: If true, the tag columns also include ModelManager's own mm: provenance (creation time, script, git state). Defaults to false, since those are rarely what you are comparing rows on. Has no effect unless tags=true.

stmtToDataFrame(stmt_str, params; db::SQLite.DB=centralDB(), is_row::Bool=false)
stmtToDataFrame(stmt::SQLite.Stmt, params; is_row::Bool=false)

Execute a prepared statement with params and return the result as a DataFrame.

tableColumns(table_name::String; db::SQLite.DB=centralDB())

Return the column names of table_name in db.

tableExists(table_name::String; db::SQLite.DB=centralDB())

Return true if table_name exists in db.

tableIDName(table::String; strip_s::Bool=true)

Return the ID column name for table (e.g. "config_id" for "configs").

withTransaction(f; mode="DEFERRED", db::SQLite.DB=centralDB())

Run f inside a transaction on db, and return its value.

Batches several statements into one commit. Nested calls join the enclosing transaction rather than starting their own, so it is safe to call from a function that may itself be invoked inside one.

mode is passed to BEGIN. ModelManager uses the default everywhere: a single statement is already atomic, and an INSERT OR IGNORE against a UNIQUE constraint is self-correcting, since a losing racer's lookup finds the winner's row. "EXCLUSIVE" exists as an escape hatch for a find-or-insert that has no such constraint to fall back on — see the note in progress.md on duplicate samplings or trials rows.

Example

withTransaction() do
    for row in rows
        insert(row)
    end
end