C2 · Publication Volume 12

Bias, Leakage and False Discovery

confirmation bias, survivorship, target leakage and multiple testing

Learning goals

After this lesson, you should be able to identify cognitive and selection biases in exploration, distinguish target leakage from legitimate predictors, design spatial and temporal validation, explain why repeated searching raises false-positive risk, calculate a simple multiple-testing example and preserve negative evidence without treating untested areas as barren.

Bias is a systematic distortion in observation, analysis or decision. It does not imply misconduct or incompetence. Geological work is especially vulnerable because exposure is incomplete, known examples are clustered, labels evolve and interpretations guide where new data are collected. Controls must therefore be built into chronology, sampling, data management and review.

Cognitive and organisational bias

Confirmation bias favours observations compatible with a preferred model and explains away contradictions. Anchoring gives excessive influence to the first interpretation or score. Availability bias overweights memorable discoveries or vivid anomalies. Sunk-cost bias continues work because prior effort was large. Group conformity can suppress viable alternatives even when no person intends it.

Mitigations include prediction matrices written before results, independent alternative construction, contradiction logs, blind or masked review where practical, rotating critique roles, explicit stopping rules and decision snapshots. Diverse viewpoints help only when evidence and reasoning are recorded; authority or seniority is not a likelihood ratio.

Organisational incentives can favour target creation over target retirement. Measure learning quality as well as advances: hypotheses discriminated, data lineage resolved, search space retired under adequate tests and criteria improved prospectively. Do not reward the relabelling of failed targets as new successes.

Survivorship and observation-process bias

Known deposits and mature prospects are survivors of geology, preservation, exposure, access, historical technology and prior search preferences. They are not a random sample of all mineral systems. Training only on survivors can teach a model to recognise exposed, accessible or historically favoured settings rather than geological potential.

“Non-deposit” labels are also uncertain. An untested point is not a verified negative; a tested location may be negative only for one geometry, depth or commodity. Use positive–unlabelled reasoning, bounded background sampling or explicit confidence in negative labels. Map exploration intensity and observation opportunity.

Negative evidence requires visibility. Absence in dense exposure is different from absence under deep cover. Preserve failed tests, but attach their footprint, detection power and hypothesis level. A negative hole should not colour an entire region barren unless its geometry and inference support that scale.

Leakage and invalid validation

Chronology and spatial separation prevent future or neighbouring target information from leaking into validation
Chronology and spatial separation prevent future or neighbouring target information from leaking into validation

Target leakage occurs when a predictor contains information unavailable at decision time or derived from the target label. Examples include using follow-up drilling to validate pre-drill ranking, calculating distance to occurrences discovered after the model date, or including a map whose interpretation was redrawn around known targets. The model may appear accurate while being unusable prospectively.

Spatial leakage occurs when neighbouring observations from the same geological cluster appear in both training and validation. Random splits can test interpolation within known districts rather than transfer to new search space. Use spatial blocks, geological clusters or leave-area-out designs, and report the distance or domain separation.

Processing leakage can occur when normalisation, feature selection or threshold tuning uses the full dataset before splitting. Fit every learned transformation within each training fold. Preserve a frozen prospective set where possible. A model revised after seeing validation performance requires a new untouched test or honest relabelling as development evidence.

Multiple testing and false discovery

When many elements, transformations, thresholds, window sizes and target cells are searched, some apparently strong patterns arise by chance under a null model. If m independent tests each use significance level \alpha, the probability of at least one false positive when all nulls are true is

$1-(1-\alpha)^m.$

For m=20 and \alpha=0.05, this is 1-0.95^{20}\approx0.642. The independence assumption is rarely exact in geoscience, but the example shows why one unadjusted low value selected from many trials is not equivalent to a predeclared test.

Family-wise error control limits the probability of any false rejection in a family. False discovery rate controls the expected proportion of false rejections among rejected hypotheses under stated assumptions. These formal methods apply to defined statistical tests, not automatically to every expert criterion. The broader controls are pre-registration, holdout evidence, transparent trial counts, correction appropriate to dependence and replication in independent geology.

Worked synthetic example

A fictional orientation study measures 12 elements and tests two size fractions, three transformations and two neighbourhood radii: 12\times2\times3\times2=144 analytical variants. The strongest variant has an unadjusted p=0.004 under a declared null model. A simple division-based family-wise threshold at \alpha=0.05 is 0.05/144=0.000347, so the selected result does not pass that conservative rule.

Suppose instead that five predeclared, scientifically distinct tests yield ordered values 0.003, 0.009, 0.018, 0.041 and 0.20. Under a step-up false-discovery-rate procedure with desired rate q=0.05, thresholds are 0.01, 0.02, 0.03, 0.04 and 0.05. The first three pass; the fourth narrowly does not. This result relies on the procedure's assumptions and on defining the test family before inspection.

Neither calculation proves a mineral system. Statistical surprise must be connected to geological process, sampling support, spatial dependence, quality control and independent validation. The 144-variant search should be recorded so that the chosen transformation is not presented as an a priori prediction.

Interpretation workflow

  1. Reconstruct what information existed at each decision date.
  2. Map exposure, access, prior search effort and label confidence.
  3. Separate confirmed positives, tested negatives and unlabelled locations.
  4. Freeze predictions, transformations and thresholds before validation.
  5. Split data by spatial or geological independence, not convenience alone.
  6. Fit preprocessing and feature selection inside training partitions.
  7. Record the full family of tried variables, windows and thresholds.
  8. Apply an error-control or replication strategy appropriate to the question.
  9. Preserve contradictions, failed targets and detection adequacy.
  10. Report prospective performance separately from retrospective fit.

Practice and review

  1. Calculate the chance of at least one false positive for 50 independent tests at \alpha=0.01.
  2. Give three examples of predictors that leak post-decision information into a target model.
  3. Design a leave-district-out validation and identify what transfer claim it supports.
  4. Explain why inaccessible areas should be unlabelled rather than negative.
  5. Create a contradiction log entry that protects an adequate barren test from later disappearance.

Sources and further reading