C2 · Publication Volume 12

Data Compilation and Source Audit

inventory, source, licence, currency, resolution and gaps

Learning goals

After this lesson, you should be able to construct a data inventory, audit source authority and licence, distinguish resolution from pixel size, test coordinate and temporal compatibility, represent null states correctly and trace a derived target criterion back to immutable source observations.

Compilation is not copying files into one folder. It is the controlled creation of an evidence graph. Each derived map must retain a link to the source object, transformation, parameters and quality decisions that produced it. A visually seamless stack can be scientifically incoherent when its layers represent different dates, supports, datums, populations or legal permissions.

Inventory before integration

Begin with a register, not a mosaic. Each object needs a stable identifier, title, data type, thematic meaning, spatial and temporal extent, coordinate reference system, vertical datum where relevant, scale or support, resolution, acquisition date, release date, source, licence, lineage, quality status, format, checksum and storage location.

Separate observations from interpretations. A field station, assay result and measured physical response are observations. A lithological polygon, interpreted fault, gridded anomaly and inversion body include modelling choices. Preserve both and state the dependency. A scan of a historical map is not equivalent to its vectorised interpretation; the georeferencing error and digitising rule belong in lineage.

Inventory tables should expose duplicates and supersession. Two filenames may contain the same release; one file may combine several vintages; a new version may correct coordinates without changing measurements. Checksums, release identifiers and explicit relations such as “derived from,” “supersedes” and “is alternative interpretation of” prevent silent double counting.

Source, licence, currency and authority

Source authority is question-specific. A legally current boundary must come from the applicable authoritative register, while an old geological map may remain the best observation of exposure that no longer exists. Record who or what issued the object only in metadata; institutional identity does not replace quality assessment.

Licence determines whether data may be accessed, transformed, redistributed or published. “Available online” does not mean unrestricted. Preserve the licence text or stable reference, attribution requirements, access conditions and any restriction on derivative works. If permission is unclear, quarantine the object from published derivatives until resolved.

Currency has at least four dates: when the phenomenon was observed, when data were processed, when the object was released and when it was accessed. Tenure, land status, infrastructure and environmental constraints can change; geological observations usually change only through reinterpretation or correction. Never use a cached administrative layer as though it were current without verification.

Coordinates, support and resolution

Coordinate reference systems include datum, projection, axis order and units. Vertical coordinates require a datum and sign convention. A coordinate pair without those definitions is incomplete. Transformation uncertainty can exceed a target footprint, especially when historical local grids, scanned maps and modern global systems are combined.

Pixel size is not necessarily spatial resolution. A coarse survey interpolated onto a fine grid contains finely spaced predictions, not fine observations. Sample support may be a point, interval, composite, catchment, footprint or volume. Resampling all layers to the smallest cell size creates visual alignment while preserving the least-resolved information.

Attribute compatibility matters as much as geometry. Concentrations may use different units, digestion methods, size fractions, detection limits and censored-value rules. Geophysical products may be measured, filtered, reduced or inverted. Before integration, create a comparison table and state which transformations are scientifically valid.

Provenance graph and transformation audit

A provenance graph links source observations through declared transformations to criteria and decisions
A provenance graph links source observations through declared transformations to criteria and decisions

A provenance graph is directed and acyclic for a fixed release. Source observations feed quality-control decisions, coordinate transformations, compositing, interpolation, feature extraction and criterion construction. Each node records code or method version, parameters, inputs, output checksum, creator role and time. Manual edits require the same record as automated transformations.

Do not overwrite raw values with corrected values. Store the correction as a derived field with reason and status. Preserve detection qualifiers, rejected results and quality-control flags. A replacement of “below detection” with zero destroys information and can create artificial contrasts after logarithmic transformation.

Reproducibility includes environment-independent meaning. A proprietary binary may be archived, but an open export with schema, units and coordinate definition improves long-term interpretability. Reproducibility does not require public release when rights or sensitivity prevent it; it requires that authorised reviewers can reconstruct the result within those boundaries.

Gap semantics and fitness for purpose

Use explicit states such as not acquired, not accessible, not licensed, not located, below detection, censored high, failed quality control, outside coverage, inapplicable and confirmed absent. A database null alone cannot communicate these meanings. A map legend and machine-readable code list should agree.

Fitness for purpose is evaluated against the decision. A regional magnetic grid may be fit for mapping crustal architecture but not a small target. A precise assay may be unfit if sample support is biased. A current cadastral layer may be legally relevant but geologically irrelevant. Rate each object for the intended use, not with one universal quality grade.

Gap analysis should identify whether the gap is reducible and whether filling it could change action. Some gaps are tolerable because other evidence dominates; others make a criterion indeterminate. Avoid acquiring data merely to fill a blank map when the result cannot discriminate hypotheses.

Worked synthetic example

A fictional target footprint is predicted to be 150 m wide. A geochemical point set has estimated positional standard uncertainty of 120 m after reconstructing a historical grid. A geophysical anomaly has 160 m positional uncertainty because its datum is incompletely documented. If the errors are treated as independent and approximately orthogonal, combined standard uncertainty is

$u=\sqrt{120^2+160^2}=200\ \text{m}.$

The uncertainty exceeds the predicted footprint. Apparent coincidence between the two layers cannot support a target-scale causal link. Regridding both products to 25 m cells would not improve this constraint. The defensible action is to resolve coordinate lineage or reacquire control, not to calculate a more detailed overlap score.

A second audit finds that the geochemical layer represents 100 m composite catchments and the geophysical grid has effective resolution about 400 m despite 50 m output pixels. The combined model must operate at a scale justified by the coarser response and positional uncertainty. The 25 m target polygons can remain as hypotheses, but their ranking should be marked indeterminate rather than precise.

Interpretation workflow

  1. Register every source and derived object with a stable identifier and checksum.
  2. Separate observation, interpretation and administrative layers.
  3. Record licence, access condition, dates and authoritative use.
  4. Validate horizontal and vertical coordinate definitions and units.
  5. Document observation support, effective resolution and detection limits.
  6. Harmonise attributes only through declared, reversible transformations.
  7. Encode distinct null, censoring, exclusion and quality-failure states.
  8. Build a provenance graph from source to target criterion.
  9. Evaluate fitness for each intended decision and scale.
  10. Quarantine unresolved objects rather than letting them silently influence ranking.

Practice and review

  1. Explain why converting a 500 m grid to 10 m pixels does not create 10 m resolution.
  2. Design null codes for a geochemical table with below-detection, missing, rejected and outside-domain records.
  3. List the metadata required to transform a historical local grid into a modern projected system.
  4. Draw a provenance graph from field sample to anomaly polygon, including every quality and transformation step.
  5. Decide whether a restricted but high-quality dataset can be used in a public target map and identify the unresolved permission question.

Sources and further reading