E2 · Publication Volume 24

Provenance, Lineage and Processing Manifests

source, transform, parameters, checksums and dependencies

Learning objectives

  • Explain why source, transform, parameters, checksums and dependencies require explicit semantic modelling.
  • Design identities, relations and constraints that preserve provenance, lineage and processing manifests across exchange.
  • Separate hard release gates from diagnostic metrics and interpretation choices.
  • Produce a reproducible processing manifest and dependency graph with validation evidence from synthetic evidence.

The lesson is complete only when the learner can defend both the model and the release decision. A neat schema without evidence, tests or declared limitations is an unverified design. The assessed artefact must make assumptions visible and distinguish source assertions from derived conclusions.

Decision context

Capture provenance at the granularity required by risk. Dataset-level lineage may be sufficient for a simple immutable copy; a filtered or corrected result may require record- or field-level derivation. The chosen granularity and its limits must be declared.

Start with a decision record: name the intended use, the evidence required, the consequence of error, the accepted uncertainty and the role authorised to accept residual risk. Then ask whether the proposed model can answer the decision question without relying on filename conventions, row order, undocumented defaults or someone’s memory. This prevents technology selection from concealing a missing semantic requirement.

The same record may be fit for one use and unfit for another. A rapid exploratory view can tolerate conditions that a released exchange package cannot. Fitness is therefore stated against a use, contract version and quality gate rather than attached permanently to the data.

Core concept

Provenance explains how a claim or dataset came to exist; lineage follows dependencies through transformations; a processing manifest captures the exact inputs, code, parameters, environment, outputs and checks needed to reproduce one run. Together they turn a result into inspectable evidence.

The working scope is source, transform, parameters, checksums and dependencies. For each item in that scope, distinguish the thing itself, the label used by a source, the claim made about it and the record that carries the claim. Identity is not a display name; a value is not its unit; an observation is not a model; current is not the same as valid. These distinctions create explicit places for correction, uncertainty and competing interpretations.

A good semantic design can be explained as a set of sentences before it is encoded. Each sentence identifies a subject, a property or relationship, an object or result, and the context under which the claim holds. Physical tables and files are then projections of those sentences, not their source of meaning.

Semantic model

Represent immutable entities, transformation activities and responsible roles as a directed acyclic derivation graph. A run records start and end, executable identity, version, parameters, input identities and fingerprints, output identities and fingerprints, environment, logs and validation results. Retries receive distinct run identities.

Test every proposed record against seven questions: What has identity? What type is it? Which property or relationship is asserted? Which spatial and temporal context applies? Which state or qualifier modifies the assertion? Which evidence supports it? Which version and activity produced the stored representation? Missing answers become explicit contract gaps.

Normalisation is used to separate independent facts, not to maximise the number of tables. A compact nested object can be semantically sound if the same identities, constraints and provenance remain explicit. Conversely, a highly normalised database can still be ambiguous when relationships and units exist only in documentation or application code.

Constraints and invariants

| Invariant | Executable or review test | | --- | --- | | Inputs and outputs are immutable versions | A manifest never points ambiguously to latest. | | Every transform has an identity | Record code or executable version plus configuration. | | Fingerprints cover bytes | Hash exact artefacts and record the algorithm. | | Validation is part of the run | Store checks, results and release decision with the activity. |

An invariant is a condition that must remain true across storage, export, correction and reprocessing. Implement it as close to the authoritative boundary as practical and repeat the check at exchange boundaries. Record rule identifier, version, severity, evaluated scope, observed value and outcome so a failure can be reproduced.

Hard gates protect identity, semantic validity, required provenance and authorised use. Diagnostic checks reveal unusual values or patterns but require interpretation. Never convert a diagnostic threshold into deletion or correction without a reviewed rule and preserved source evidence.

Quantitative reasoning

Lineage completeness can be measured as C_l = n_e / n_r, where n_e is the number of required dependency edges present and n_r is the number required by the run contract. Reproduction compares output fingerprints and semantic checks; byte inequality may be acceptable only when a declared non-determinism policy supplies equivalence tests.

Every reported ratio states its numerator, denominator, exclusions and evaluation time. Stratify results by source, entity type, contract version or processing run where aggregation could hide a local failure. Counts accompany percentages so a seemingly large change based on a tiny denominator remains visible.

Precision is part of meaning. Do not add decimal places merely because a storage type permits them, and do not round identity, interval or coordinate fields without a declared tolerance and test. Quantitative summaries support a release decision; they do not replace semantic review.

Evidence and uncertainty

A checksum proves byte identity under the stated algorithm, not correctness, authorship or trustworthiness. Pair fixity with schema validation, domain checks and review. Logs alone are weak provenance because they may be incomplete, unstructured or detached from exact input and output identities.

Build an evidence packet containing preserved source reference, acquisition or assertion context, applicable method, validation results, reviewer decision and links to derivatives. Classify uncertainty as observational, semantic, structural, parametric or policy-related where that distinction changes treatment. “Unknown” is a valid state when the evidence cannot justify a stronger claim.

Contradictory evidence remains available. The model may select one current assertion, but the reason, competing assertion and effective time are retained. This makes later reinterpretation possible without pretending the earlier evidence never existed.

Interfaces and storage

Expose manifests as machine-readable records and provide a human-readable run summary derived from the same source. Use stable identifiers for runs, jobs, datasets and facets. Cross-system lineage preserves external identifiers and namespace rather than replacing them with local row numbers.

Design an interface from the logical contract outward. Specify identifiers, types, cardinalities, units, value states, coordinate and time references, version negotiation, validation behaviour and structured errors before choosing a serialisation. The physical representation then declares its mapping to those logical elements.

Storage optimisation may partition, compress, index or cache data, but it must not change identity or silently remove context. A derived representation points to immutable inputs and a processing manifest. A cache carries freshness and contract-version information and is never treated as the only evidence copy.

Governance and access

Set minimum manifest fields by processing class and decision consequence. Release gates verify that all required dependencies resolve, fingerprints match, validation passed and sensitive parameters are handled appropriately. Provenance records are retained at least as long as any dependent release.

Governance is expressed through named roles, review states and versioned decisions, not through references to a particular organisation. Define who may propose, validate, approve, supersede and withdraw each governed resource. The audit trail records the role and event while avoiding unnecessary personal data.

Apply least-necessary access to source evidence and derivatives. Access controls must not erase identifiers, lineage or quality metadata needed to understand an authorised release. When policy is unresolved, quarantine the output with a precise reason and escalation route.

Integration checkpoint

Immutable entities and activities form an auditable lineage graph
Immutable entities and activities form an auditable lineage graph

The diagram summarises the control flow for this lesson. Read it from source evidence through semantic structure and validation to a decision-ready artefact. Each arrow should correspond to a declared relationship or transformation; each boundary should have a contract; each released node should have an identity, version and provenance pointer.

Integrate the lesson by adding a reproducible processing manifest and dependency graph with validation evidence to the evolving synthetic data package. Verify that earlier artefacts still resolve and that the new model does not overwrite observations, identifiers, values or versions introduced in previous lessons. Record every changed assumption.

Synthetic worked example

A synthetic pipeline normalises sample identifiers, converts units and joins results to locations. The learner fingerprints two inputs, records code version and parameters, emits validation counts, fingerprints the output and draws the derivation graph. A changed vocabulary version produces a new run and output even when row count stays constant.

Work the example in four passes:

  1. Preserve the received records and write the intended decision without correcting anything.
  2. Identify entities, claims, context, uncertainties and policy constraints; mark every unresolved item.
  3. Apply the versioned rules, create derivatives and record the exact transformation plus validation evidence.
  4. Issue an accept, reject or quarantine decision and show how an independent reviewer can reproduce it.

Because the example is entirely synthetic, its values demonstrate method only. The important result is the chain from received evidence to justified decision. If a required fact is absent, the worked solution records the gap rather than manufacturing a plausible value.

Practice task

Instrument a synthetic three-step transformation with manifests. Record exact identities, fingerprints, parameters, environment and checks. Reproduce the output from a clean location, compare byte and semantic results, then intentionally change one dependency and show how the graph exposes the affected outputs.

Use the following acceptance criteria:

  • All required identifiers and references resolve to declared types.
  • Every transformation preserves the received evidence and records its derivation.
  • Invalid, unknown and inapplicable states remain distinct and machine-testable.
  • The output identifies the contract, vocabulary and processing versions used.
  • A second reader can reproduce the validation result without private knowledge.

Submit the source snapshot, authored contract or model, validation output, derivative, manifest and a short decision record. A screenshot alone is insufficient because it cannot demonstrate the exact input, version or rule execution.

Common failure modes

  • A manifest points to a mutable latest input.
  • Code version is recorded but parameters are absent.
  • A checksum is presented as proof that data is correct.
  • A retry overwrites the failed run and its diagnostics.

These failures share a pattern: convenient representation is mistaken for verified meaning. Diagnose the earliest boundary at which an assumption became implicit. Correct by restoring source evidence, making the assumption a versioned field or rule, rerunning dependent transformations and superseding—not overwriting—the affected release.

Do not repair a failure by adding an undocumented default. A blocked result with a specific missing dependency is safer and more reusable than a complete-looking result whose meaning cannot be reconstructed.

Review questions

  1. How do provenance, lineage and a manifest differ?
  2. What does a checksum prove and not prove?
  3. Why should retries have distinct identities?
  4. How is lineage granularity selected?

For each answer, identify the governing invariant, the evidence needed to evaluate it and the appropriate release behaviour when the invariant fails. A strong answer distinguishes scientific uncertainty from missing semantics and distinguishes a recoverable warning from a hard contract violation.

Sources and further reading