E6 ยท Publication Volume 28

Ingestion and Validation Pipelines

landing, staging, normalisation, QC and publication

*landing, staging, normalisation, QC and publication*

Replayable landing, staging, normalisation, quality-control and publication pipeline
Replayable landing, staging, normalisation, quality-control and publication pipeline

Learning objectives

This lesson is general and institution-neutral. It uses no real company, individual, property, project or identifiable place. Generic roles describe responsibilities only, and SYN-ARCH identifiers denote explicitly synthetic teaching evidence.

  • Frame the decision governed by landing, staging, normalisation, QC and publication.
  • Model the relevant boundaries, states and contracts before selecting an implementation.
  • Define measurable invariants, failure evidence and a safe release consequence.
  • Produce a replayable ingestion contract with quarantine and publication evidence from synthetic evidence and defend its trade-offs.

Decision boundary

Decide whether a received package is structurally readable, semantically interpretable, fit for a stated use and eligible for publication. These are separate gates. Preserve the received bytes and manifest before parsing so later corrections can be replayed against the same evidence. Validation should reject or quarantine a bounded object with an explanatory finding, not silently repair source meaning. Normalisation creates a new representation and records every conversion. Publication selects an accepted version and makes its status visible; it does not erase rejected records or convert uncertainty into apparent completeness.

Core concepts

Landing is immutable receipt; staging is a controlled workspace; normalisation maps declared source semantics to canonical semantics; quality control evaluates rules and measures; publication promotes an identified output. A pipeline is a state machine rather than a chain of ad hoc scripts. Every transition has preconditions, produced artefacts, findings and terminal outcomes. Row-level validity, file-level integrity and package-level completeness differ. Fatal errors, quarantinable findings, warnings and informational observations need governed meanings. Idempotency ensures that replaying the same input and pipeline version does not create duplicate identities or divergent outputs.

System model and contracts

A receipt manifest records package identity, source role, receipt time, media types, byte counts, digests and expected members. Parsers emit typed staging records plus parse findings. Normalisers require explicit unit, reference, code-list and null policies and emit transformation events. Validators produce findings with rule identifier, severity, object path, observed value, expected condition and evidence. Publication creates a new immutable data-product version referencing accepted records, excluded records, rule set, software build, parameter set and reviewer decision. Quarantine remains addressable and access controlled so correction can link to the rejected predecessor.

Invariants and acceptance criteria

| Invariant | Test evidence | Release consequence | |---|---|---| | Received bytes and their manifest remain immutable and addressable. | contract test and recorded counterexample | block publication | | Every transformation emits provenance and never silently changes meaning. | replay comparison and digest check | quarantine the artefact | | Publication references explicit accepted, excluded and unresolved populations. | role-based acceptance trace | return the decision unresolved | | Replaying identical versions yields an identical output manifest. | failure injection and recovery record | retain the last verified version | | Quarantine preserves evidence without granting publication status. | domain review against declared evidence | record an explicit review finding |

Quantitative engineering

For N received records, report accepted A, quarantined Q, rejected R and unresolved U with A+Q+R+U=N at the declared counting support. Completeness for required field j is C_j=n_{present,j}/N_j, but present is not equivalent to valid. Duplicate rate, reference-resolution rate, unit-conversion count, rule execution coverage and finding age reveal different risks. Reproducibility compares output manifests and content digests under the same input, pipeline, rule and parameter versions. Throughput is reported with object-size distribution and back-pressure behaviour, not records per second alone.

Data quality, evidence and uncertainty

Quality rules trace to product specifications, domain constraints, accepted code lists, reference registries and observed failure cases. A rule records applicability and cannot judge records outside its support. Keep raw, normalised and published states distinct so a reviewer can inspect whether a surprising value came from source, parsing, conversion or later interpretation. Sampling may estimate quality only when the sample design and uncertainty are reported. Absence of findings means only that executed rules found none; it is not proof that the product is fit for every use.

Interoperability and versioning

The ingestion contract declares package media types, required members, naming and identifier rules, schema editions, units, spatial and temporal references, missing-value encoding, character encoding, code lists, digest algorithm and maximum limits. Findings use a versioned schema and stable rule identifiers so trends remain comparable. Additive optional fields may be compatible; changed meaning, unit or reference is not. Pipeline outputs are immutable and content addressed where practical. A correction is a new receipt linked by revision or derivation, never an overwrite that destroys the evidence reviewed earlier.

Security and professional responsibility

Treat received content as untrusted. Enforce size, nesting, compression, member-count and geometry limits before expensive parsing. Isolate parsing and transformation privileges from publication privileges. File names and embedded paths cannot choose storage locations. Scan active content according to risk, reject unexpected executable forms and never log secrets or full sensitive rows merely to explain an error. The receipt identity and submitting role are authenticated; authorisation is checked again at publication. Audit events record policy, rule and reviewer versions without exposing restricted content to broader telemetry.

Operational workflow and observability

A pipeline run has immutable input reference, run identity, pipeline build, configuration digest, start and finish times, stage states, counters, findings and output manifest. Retries resume from a safe checkpoint or rerun an idempotent stage; they do not skip failed validation. Back-pressure limits admission when staging or review queues exceed policy. Dashboards separate technical failure from data-quality rejection. Replay tests run old accepted and rejected packages against a candidate pipeline, compare changed findings and require explanation. Retention policies preserve evidence long enough to reproduce published versions and investigate incidents.

Integration checkpoint

Connect the ingestion and validation pipelines artefact to the preceding volume architecture. Trace one synthetic object from source identity through the new boundary to a reviewed output, then trace one rejection or failure back to the earliest violated invariant. Update the architecture decision record with the chosen option, alternatives, assumptions, evidence, consequences, owner role, review state and triggers for reconsideration. A checkpoint passes only when another reviewer can reconstruct both the successful path and the blocked path without oral explanation.

Synthetic worked example

SYN-ARCH-03 receives a package whose interval table is syntactically valid but names depth in one field while values follow another convention. A permissive script silently flips signs and publishes. The governed pipeline instead preserves the receipt, parses successfully, fails semantic reference resolution and quarantines the affected table while retaining independent imagery. After a corrected manifest arrives, a new receipt derives from the first. Replay with the same pipeline produces an identical digest. A candidate rule change alters three warnings; the comparison report requires a reviewer to accept or reject that change before promotion.

Practice and assessment

  1. Which evidence belongs to receipt, staging, normalisation and publication?
  2. What makes a stage idempotent and safely replayable?
  3. When should an object be rejected, quarantined or left unresolved?
  4. How can a reviewer explain changed findings after a rule update?

Assessed artefact: a replayable ingestion contract with quarantine and publication evidence. Submit the artefact with its source manifest, acceptance evidence, unresolved risks and a short explanation of why one plausible alternative was not selected.

Common failure modes

  • Overwriting the received package after a correction.
  • Combining parsing, semantic validation and approval into one opaque status.
  • Silently repairing units, references or identifiers.
  • Retrying a non-idempotent stage and creating duplicate records.
  • Equating no validation findings with fitness for every use.

Sources and further reading