# Decision: add bounded full-source discovery to Ask OKF

Status: implemented for evaluation, 19 September 2026.

## Problem

A source corpus can contain thousands of pages while its governed context index
contains only one small, explicitly modelled task. Searching more pages does not
by itself supply the missing concepts, qualifications or evidence requirements.
The DWP hospital and staff questions exposed this distinction.

## Decision

Keep the existing context index and add an optional `context_corpus` manifest.
The manifest binds the existing semantic index, all non-empty source page
records and a complete sharded lexical index. It records empty extracted pages
separately. Acquisition, source publication and legal applicability remain
separate facts.

A reusable `context/corpus.ts` module retrieves a bounded set of whole candidate
pages, verifies every transferred and decoded file, then calls the same context
assembler. It does not add relationships, aliases, evidence requirements or an
AI answer. The browser, WebMCP adapter and remote MCP service use this module.
The package has an optional `retrieval` section. Existing packages without this
section remain compatible.

Page ranking uses the question's words only, after the documented normalisation
and generic question-word exclusion. It does not expand the query using matched
concept labels: a broadly declared alias might name a more narrowly scoped rule.
Concept resolution and directed traversal remain separate, inspectable results.
Whole candidate pages are prioritised before those graph seeds, so a large graph
cannot silently consume the entire package before new evidence is included.

## Bounds and security

Each manifest/file is at most 4 MiB decoded. References must remain beneath the
manifest directory, with no parent paths, encoded escapes, arbitrary URLs,
credentials or redirects. Both compressed and decoded SHA-256 hashes and exact
lengths are checked. Streaming reads enforce the declared limits before parsing.

Per call, discovery allows 24 distinct query tokens, 16 candidate pages, 64 files,
16 MiB transferred and 32 MiB decoded. Tokens over 64 characters are omitted with
an explicit query-budget report. Pages are ranked by the sum of
`1 + floor(1000 * log2(1 + corpus_records / posting_count))` for each matched term;
ties use the canonical record ordinal. This is lexical ranking, not a confidence
score. Record identifiers and source text do not encode executable instructions.

The working semantic/evidence index remains within 4 MiB; whole candidate pages
are omitted if necessary. The existing node, edge, depth and response-byte limits
then apply. All query, candidate, resource, working-index and response omissions
are visible and make the result insufficient. A caller can still get useful
source evidence in an insufficient package. There is no web-search or model
fallback.

The public service has a separately approved immutable corpus version. Only its
manifest-declared files at that version can be fetched. It caches verified public
bytes within a bounded cache; cache hits do not change package accounting or
identity. The earlier fully embedded index remains available by explicit version.
No caller-supplied URL, credentials or write-capable tool is accepted.

## Consequences

Full-source discovery is available before full domain modelling. This makes
missing semantic coverage measurable without claiming it is solved. Results may
include historical amendments, repeated passages, unrelated neighbouring text
or damaged PDF extraction. Existing broad aliases can resolve to narrowly scoped
concepts; consumers must read the scope. No completeness claim follows from a
lexical match, recent capture, official source label or graph connection.

The DWP discovery corpus deliberately has no completeness profiles. Its packages
remain insufficient until independently reviewed task requirements establish a
bounded question scope. The original custody profile remains separately usable
within its original assumptions.

## Validation

Synthetic non-DWP tests cover deterministic whole-page retrieval, normalisation,
unsafe paths, altered compressed data, decompression limits, invalid ordinals,
conflicting canonical evidence and small output budgets. Browser tests cover
hash/snapshot rejection, source links, visible retrieval limits and unchanged
narrow-index journeys across Chrome, Firefox and WebKit. Source producers verify
all indexed pages; evaluation reports independently located evidence separately
from answerability and specialist acceptance.