E7 · Publication Volume 29

Retrieval and Evidence Selection

recall, precision, source quality and temporal relevance

*recall, precision, source quality and temporal relevance*

Retrieval funnel from query decomposition through source-quality and temporal gates
Retrieval funnel from query decomposition through source-quality and temporal gates

Learning objectives and boundary

This lesson is general and institution-neutral. It uses no real company, individual, property, project or identifiable place. Generic roles describe responsibilities only, and every SYN-AI identifier denotes explicitly synthetic teaching evidence.

  • Frame the decision governed by recall, precision, source quality and temporal relevance.
  • Separate generated proposals from admissible evidence, deterministic results and accountable judgement.
  • Define measurable failure, abstention, escalation and release conditions.
  • Produce a versioned retrieval benchmark and evidence-admission register from synthetic evidence and defend its controls.

Decision and professional boundary

Retrieval decides what evidence the reasoning stage is allowed to see. Separate candidate discovery from evidence admission. Discovery should seek broad recall across terminology, identifiers, spatial extents and document structures. Admission should test authority for the intended claim, source version, temporal validity, independence, directness and access status. Relevance alone is insufficient: a highly similar superseded interpretation may be worse evidence than a less similar current observation.

The search question must be decomposed into claim-sized evidence needs. Queries for a unit definition, a measured value, a method limitation and a competing interpretation should not share one unexamined ranking. Preserve rejected candidates and rejection reasons so missing evidence can be distinguished from an overly narrow query or an admission rule that is too strict.

Core concepts

| Concept | Operational meaning | |---|---| | Candidate recall | The fraction of relevant evidence found before admission and ranking. | | Admission precision | The fraction of admitted items that are fit to support the specific claim. | | Temporal validity | The interval during which an observation, product or interpretation is applicable. | | Evidence directness | Whether a source contains the observation itself or only repeats a derived statement. | | Independence | Whether apparent corroboration arises from distinct acquisition and reasoning paths. |

Evidence model

A retrieval run is an immutable result set containing query decomposition, corpus snapshot, filters, ranking configuration, candidate locators, scores, lineage groups, admission outcomes and missing-evidence declarations. Evidence selection is claim specific: the same source may be admissible for a historical description but inadmissible for a current coordinate or approved status.

Controlled workflow

  1. Decompose the request into atomic evidence needs and unknowns.
  2. Generate lexical, identifier, spatial, temporal and structural queries.
  3. Retrieve broadly and record ranks, scores and query versions.
  4. Deduplicate by source lineage rather than wording alone.
  5. Apply quality, directness, validity, independence and access gates.
  6. Return admitted evidence, rejected candidates and declared gaps.

Every step emits a versioned artefact or an explicit failure. A later stage consumes only verified outputs from the preceding stage; conversational context is never an undocumented data channel.

Measures and acceptance criteria

Use judged synthetic queries to report recall, precision, rank-sensitive utility and gap-detection accuracy. Evaluate source-lineage diversity separately so many copies of one statement do not look like corroboration. Thresholds are fixed before the test set is inspected. When complete relevance judgements are unavailable, report judgement coverage and do not present partial recall as absolute recall.

$P=\frac{TP}{TP+FP},\qquad R=\frac{TP}{TP+FN},\qquad F_1=\frac{2PR}{P+R}$

| Gate | Required evidence | Release consequence | |---|---|---| | Identity | Resolvable claim, source, configuration and result IDs | Block when any identity is ambiguous | | Grounding | Every factual claim reaches sufficient source spans | Remove or withhold unsupported claims | | Independence | Lineage and partition checks show no circular or target information | Invalidate affected support and scores | | Review | Required roles disposition the exact candidate version | Keep the candidate non-published | | Reproducibility | Manifest, tools and checks reconstruct the evidence package | Return the package for correction |

Claim–source and system contract

The retrieval contract declares corpus snapshot, query version, admissible media, required source states, time reference, spatial extent, top-k limits, reranking method, deduplication key and gap semantics. Every selected item carries its original rank and admission explanation; generation cannot silently substitute an unrecorded source.

Security, privacy and access boundary

Apply access filters before ranking and again before context assembly. A ranking score must not reveal the existence of a restricted item to an unauthorised requester. External retrieval is disabled unless the task contract names allowed endpoints, evidence capture rules and versioning behaviour. Retrieved text remains untrusted content and cannot alter tool permissions.

Human review and escalation

Review false negatives and false positives by evidence class, not only by query. A reviewer inspects why a current observation was missed, why a derived summary outranked it and whether lineage deduplication worked. Changes to corpora, embeddings, tokenisation or ranking create a new retrieval version and require regression evaluation.

Worked synthetic example

A synthetic query asks for evidence about a contact near grid cell G-17 during interpretation snapshot 4. Lexical retrieval finds an old summary with the exact phrase, while spatial retrieval finds two current observations and a later interpretation using different vocabulary. Admission rejects the old summary for current-status support but retains it as historical context. Two summaries copied from one memo are grouped as one lineage. The result exposes one direct observation, one independent observation, one current interpretation and one unresolved gap in orientation data.

Counterexample and failure analysis

Selecting the five most similar passages and asking a model to “use the best sources” hides every important decision. Similarity is not authority, five is not a completeness argument, and duplicated derivatives can dominate the context. Without a corpus snapshot and rejected-candidate record, later reviewers cannot tell whether the evidence changed or the reasoning did.

Frequent failure modes

  • Optimising similarity without evidence admission
  • Counting copied sources as independent support
  • Mixing current and superseded states
  • Reporting recall without judgement coverage
  • Allowing retrieval results to change tool permissions

Practical exercise

  1. Write four evidence needs for one geological question.
  2. Build a judged set containing direct, derived, obsolete and inaccessible candidates.
  3. Calculate precision, recall and F1 for two retrieval configurations.
  4. Explain how lineage grouping changes the apparent support count.

Assessed artefact

Submit a versioned retrieval benchmark and evidence-admission register with its source manifest, configuration versions, acceptance evidence, rejected cases, unresolved risks and a short explanation of why one plausible alternative control was not selected. The artefact is incomplete if its diagrams or prose cannot be reconciled with machine-checkable evidence.

Verification checkpoint

Trace one released synthetic claim backwards through review, confidence, verification, evidence graph, tool result or source span to an immutable input. Then trace one rejected claim to the earliest failed contract. Re-run the package from its manifest and compare content digests. The checkpoint passes only when a second reviewer can reconstruct the accepted and blocked paths, identify every assumption and reproduce the release decision without access to the original conversation. Record unresolved risk instead of completing a missing fact with generated text.

Sources