E6 ยท Publication Volume 28

Testing Strategy

unit, property, golden-data, visual, integration and domain review

*unit, property, golden-data, visual, integration and domain review*

Layered test portfolio linking properties, golden data, visual evidence, integration and domain review
Layered test portfolio linking properties, golden data, visual evidence, integration and domain review

Learning objectives

This lesson is general and institution-neutral. It uses no real company, individual, property, project or identifiable place. Generic roles describe responsibilities only, and SYN-ARCH identifiers denote explicitly synthetic teaching evidence.

  • Frame the decision governed by unit, property, golden-data, visual, integration and domain review.
  • Model the relevant boundaries, states and contracts before selecting an implementation.
  • Define measurable invariants, failure evidence and a safe release consequence.
  • Produce a risk-linked test portfolio with controlled evidence corpora from synthetic evidence and defend its trade-offs.

Decision boundary

Select tests from decisions, risks, invariants and failure consequences. Unit tests isolate deterministic logic; property tests examine statements that should hold across many inputs; controlled corpora preserve known examples and counterexamples; visual tests detect representation changes; integration tests exercise real boundaries; end-to-end tests follow a user decision; domain review judges scientific meaning that software cannot determine alone. No layer replaces another. A passing suite supports a bounded claim for declared versions and environments, not universal correctness.

Core concepts

A test oracle states how correctness is judged. Exact expected values work for deterministic examples; invariants, metamorphic relations, conservation rules and tolerances serve when an exact result is unavailable or inappropriate. Controlled golden data is versioned evidence with origin, purpose and expected behaviour, not an unexplained output copied after a failure. Visual comparison separates structural, geometric, colour and text changes and requires accessibility checks beyond pixel matching. Domain review records the question, evidence inspected, reviewer role, disagreement and disposition instead of becoming an undocumented approval click.

System model and contracts

Maintain a trace matrix from requirement and risk to invariant, test level, fixture, oracle, environment, result and release consequence. The corpus includes minimal normal cases, boundaries, malformed inputs, missing values, mixed references, duplicate identities, extreme magnitudes, topology defects, permission changes, partial failures and historical versions. Each fixture has stable identity, manifest, generation or observation source, license or use boundary, expected findings and sensitivity class. Test results record exact code, contract, dependency and data versions plus artifacts needed to reproduce a failure.

Invariants and acceptance criteria

| Invariant | Test evidence | Release consequence | |---|---|---| | Every active requirement and high-consequence risk traces to adequate test evidence. | contract test and recorded counterexample | block publication | | Every fixture declares identity, origin, purpose and expected behaviour. | replay comparison and digest check | quarantine the artefact | | Golden expectations change only through reviewed evidence. | role-based acceptance trace | return the decision unresolved | | Tolerance and visual comparisons preserve declared scientific semantics. | failure injection and recovery record | retain the last verified version | | Domain review findings remain linked to the exact candidate release. | domain review against declared evidence | record an explicit review finding |

Quantitative engineering

Requirement coverage is C_r=n_{tested}/n_{applicable}, while risk-weighted coverage is C_w=(w_1I_1+w_2I_2+...+w_nI_n)/(w_1+w_2+...+w_n) with declared weights and a binary or graded evidence indicator. Report mutation or fault-detection evidence, property-case distribution, flaky rate, visual-diff class, boundary coverage and unresolved review findings. Line coverage alone says which code executed, not whether scientific meaning was checked. Tolerance tests state absolute, relative, spatial and topological components and examine values just inside and outside the boundary.

Data quality, evidence and uncertainty

Test evidence remains trustworthy when failures cannot be erased by regenerating expectations without review. Changes to a golden result require reason, before-and-after comparison, affected invariant and independent disposition. Separate synthetic fixtures from observed or curated reference material and never present generated values as field truth. Property generators record seeds and shrinking history for counterexamples. Visual baselines retain viewport, rendering settings, fonts, colour interpretation and reference data. Integration environments use real contract implementations and storage boundaries; substitutes are labelled and cannot close the corresponding production risk.

Interoperability and versioning

A test case contract defines preconditions, fixture version, action, oracle, tolerance, expected evidence and cleanup. Test-result schemas distinguish pass, fail, error, skipped, quarantined and inconclusive; only a justified policy maps them to release decisions. Interfaces provide deterministic hooks for time, random sources and external dependencies where reproducibility requires control, without changing production semantics. Visual and data snapshots use content digests and reviewed update workflows. Domain review forms use controlled findings and free explanation, and link every disposition to the exact candidate release.

Security and professional responsibility

Test authorisation boundaries with least-privilege identities and negative cases. Fixtures contain no unapproved sensitive data, live credentials or executable malicious payloads; hostile structures are isolated and size bounded. Test logs and failure attachments inherit data classification and are redacted before broad distribution. Supply and build evidence verifies the test tools and dependencies used to accept a release. A test-only bypass cannot be enabled in a production path, and debug endpoints, seeded accounts or relaxed policy are detected by configuration tests.

Operational workflow and observability

Use release gates tied to consequence: one high-risk invariant can block release even when thousands of low-risk tests pass. Quarantine flaky tests with owner role, reason, risk, compensating evidence and expiry rather than silently retrying until green. Production signals feed new counterexamples into a reviewed corpus. Periodically prove that tests detect seeded faults and that retired tests no longer trace to active obligations. Archive results with the release manifest so an investigation can reconstruct what was known, failed, waived and reviewed at publication time.

Integration checkpoint

Connect the testing strategy artefact to the preceding volume architecture. Trace one synthetic object from source identity through the new boundary to a reviewed output, then trace one rejection or failure back to the earliest violated invariant. Update the architecture decision record with the chosen option, alternatives, assumptions, evidence, consequences, owner role, review state and triggers for reconsideration. A checkpoint passes only when another reviewer can reconstruct both the successful path and the blocked path without oral explanation.

Synthetic worked example

SYN-ARCH-11 has many passing unit tests but no oracle for mixed coordinate reference during a tiled export. A synthetic boundary fixture reveals that two geometries align visually after automatic zoom yet differ in world coordinates. The portfolio adds a reference property, a round-trip corpus, a visual overlay with world-space markers, an integration export and a domain review question. A seeded axis-order fault is detected at three layers. The failure blocks publication until the contract and transform are corrected, while unrelated presentation differences remain separately classified.

Practice and assessment

  1. Which oracle is appropriate for each scientific and system claim?
  2. Which counterexamples exercise boundaries that normal samples miss?
  3. How are synthetic, curated and observed test data distinguished?
  4. Which failure or inconclusive result must block publication?

Assessed artefact: a risk-linked test portfolio with controlled evidence corpora. Submit the artefact with its source manifest, acceptance evidence, unresolved risks and a short explanation of why one plausible alternative was not selected.

Common failure modes

  • Using code-line coverage as proof of domain correctness.
  • Replacing a failed expected result without an independent review.
  • Testing only happy paths and convenient numeric ranges.
  • Using pixel equality as the sole visual and accessibility oracle.
  • Letting repeated retries hide an unstable or nondeterministic test.

Sources and further reading