E6 · Publication Volume 28

Jobs, Workflows and Reproducibility

parameters, dependencies, status, retry and outputs

*parameters, dependencies, status, retry and outputs*

Workflow dependency graph with job states, retries, checkpoints and immutable outputs
Workflow dependency graph with job states, retries, checkpoints and immutable outputs

Learning objectives

This lesson is general and institution-neutral. It uses no real company, individual, property, project or identifiable place. Generic roles describe responsibilities only, and SYN-ARCH identifiers denote explicitly synthetic teaching evidence.

  • Frame the decision governed by parameters, dependencies, status, retry and outputs.
  • Model the relevant boundaries, states and contracts before selecting an implementation.
  • Define measurable invariants, failure evidence and a safe release consequence.
  • Produce a replayable workflow manifest and job-state contract from synthetic evidence and defend its trade-offs.

Decision boundary

Model a job as a bounded transformation with declared inputs, parameters, environment, code identity, policy and expected outputs. Model a workflow as a dependency graph whose edges carry artefacts and completion conditions. Submission, scheduling, execution, approval and publication are different responsibilities. A retry is safe only when repeated execution cannot silently duplicate side effects or replace a verified output. Long calculations need checkpoints whose validity is tied to the same inputs and execution contract. The system reports uncertain and partial outcomes explicitly rather than converting process completion into scientific acceptance.

Core concepts

Use a finite state model such as accepted, queued, running, awaiting review, succeeded, failed, cancelled and superseded, with allowed transitions and terminal meanings. Separate execution success from output validation and publication. Idempotency means the same logical request produces one declared effect, not necessarily that it consumes no resources twice. Retries classify transient, permanent, policy and data errors. A workflow graph is acyclic for one run even when an outer schedule repeats it. Dynamic fan-out records the discovered tasks, ordering rule and dependency evidence so replay does not rely on an invisible queue state.

System model and contracts

A workflow manifest records workflow identity and version, run identity, requester role, submitted time, input artefact identities and digests, normalized parameters, task graph, execution image or environment identity, resource class, retry policy, checkpoints, expected output contracts, validation gates and retention. Each event appends state, time, actor or worker identity, attempt, correlation identity and reason. Outputs first enter a candidate area, receive digests and validation results, then become immutable published artefacts through an atomic manifest update. The manifest can be exported with all non-secret information required to replay the run.

Invariants and acceptance criteria

| Invariant | Test evidence | Release consequence | |---|---|---| | Every run declares immutable inputs, normalized parameters and workflow revision. | contract test and recorded counterexample | block publication | | State transitions follow the declared job-state contract. | replay comparison and digest check | quarantine the artefact | | Retries cannot create duplicate published effects. | role-based acceptance trace | return the decision unresolved | | Execution success remains separate from validation and approval. | failure injection and recovery record | retain the last verified version | | Published outputs are linked through one complete immutable manifest. | domain review against declared evidence | record an explicit review finding |

Quantitative engineering

For a workflow graph, critical-path duration is the longest dependency-weighted path rather than the sum of all task durations. Report queue, execution, validation and publication delay separately. Retry amplification is R_t=n_{attempts}/n_{jobs}, accompanied by error class and eventual outcome. Measure task success by input and version cohort, checkpoint recovery distance, duplicate-effect count, orphan output count and provenance completeness. Reproducibility compares output digests when exact determinism is promised, or declared scientific tolerances and structural invariants when parallel or numerical execution permits bounded variation.

Data quality, evidence and uncertainty

Reproduction evidence includes immutable inputs, normalized parameters, task graph, transformation identity, environment declaration, random seed or stochastic policy, logs, validation findings and output manifest. Capture external reference data as versioned inputs rather than assuming it will remain available or unchanged. Failure injection covers worker loss, timeout, duplicated delivery, late event, partial output, unavailable dependency and corrupted checkpoint. A replay report distinguishes exact, tolerance-equivalent, structurally equivalent and non-reproducible outcomes and explains which source of variation was expected.

Interoperability and versioning

The job contract defines submission schema, canonical parameter representation, input version rules, accepted state, progress semantics, cancellation behaviour, retry classes, result links and problem responses. A client polls or subscribes using stable run identity rather than inferring state from filenames. Completion responses link to immutable output manifests and validation status. Workflow revisions receive new identities or declared versions; a historical run always points to the exact revision used. Checkpoints are private implementation artefacts unless their portability, schema and validity are explicitly contracted.

Security and professional responsibility

Authorise submission, observation, cancellation, approval and output retrieval separately. The execution identity receives only the inputs and destinations needed for its task and cannot approve its own result. Parameter validation prevents path, command, expression and resource-injection attacks. Secrets are referenced through controlled runtime mechanisms and never embedded in manifests, logs or checkpoints. Limit CPU, memory, storage, graph size and execution duration by workload class. Audit records distinguish the requesting role, approving role and execution identity while minimizing personal data.

Operational workflow and observability

Monitor state-transition age, queue depth by resource class, active attempts, retry reason, checkpoint health, dependency wait, output validation and orphan candidates. A reconciler compares manifests with execution and storage state so lost events do not leave a run permanently ambiguous. Cancellation is cooperative where possible and records whether side effects remain. Recovery replays idempotent transitions from retained events. Changes to workflow code, base environment, dependency or policy trigger a controlled replay corpus before new runs are accepted, and historical runs remain interpretable after the old executor is retired.

Integration checkpoint

Connect the jobs, workflows and reproducibility artefact to the preceding volume architecture. Trace one synthetic object from source identity through the new boundary to a reviewed output, then trace one rejection or failure back to the earliest violated invariant. Update the architecture decision record with the chosen option, alternatives, assumptions, evidence, consequences, owner role, review state and triggers for reconsideration. A checkpoint passes only when another reviewer can reconstruct both the successful path and the blocked path without oral explanation.

Synthetic worked example

SYN-ARCH-07 creates surface tiles in parallel. A worker times out after writing output but before reporting success, so an automatic retry writes a second object and the workflow publishes both. The revised design assigns one logical task identity, writes attempt outputs to candidate locations and promotes only the validated digest referenced by the manifest. A killed worker resumes from a compatible checkpoint. Replaying the same inputs and parameters yields equivalent geometry within the declared tolerance, one published member per task and a complete state-event history.

Practice and assessment

  1. Which identities and artefacts are sufficient to replay this run?
  2. Which failure classes are safe to retry and why?
  3. How are execution, validation, approval and publication separated?
  4. What evidence defines equivalent output when exact digests may differ?

Assessed artefact: a replayable workflow manifest and job-state contract. Submit the artefact with its source manifest, acceptance evidence, unresolved risks and a short explanation of why one plausible alternative was not selected.

Common failure modes

  • Calling a script reproducible without retaining inputs and environment identity.
  • Retrying every failure without classifying permanence or side effects.
  • Treating process exit as scientific output acceptance.
  • Publishing files before the complete manifest and validation are atomic.
  • Inferring job status from storage paths or log text.

Sources and further reading