OKF Explorer evaluation harness

The encoded record route sharing change has fresh local Heritage question and journey observations, run on 29 October 2026 and bound to its exact app build. The previous receipt and both compressed result files are preserved byte for byte under validation/encoded-route-sharing/2026-10-28/pre-update/, with a SHA-256 inventory. These local observations do not establish public deployment or domain-specific policy acceptance.

The producer-declared catalogue report links change also has fresh local Heritage observations, bound to Explorer tree e1aad43dc8236e387654c86d67f1575ebdd4fd94256ea1e9628a879829e34c4a. All 100 questions scored at least 80, with a 92.6 average, and all three local journeys passed. The previous receipt and compressed results are preserved byte for byte under validation/catalogue-report-links/2026-10-29/pre-update/, with a SHA-256 inventory. These local observations do not establish public deployment or domain-specific policy acceptance.

This repository now carries repeatable browser evaluation suites for the UK Government APIs OKF pack, the hosted GOV.UK CKAN OKF pack, and future large OKF bundles.

The suite evaluates three publication goals:

Evidence tied to an Explorer build

The Heritage local candidate receipt binds its browser results to an exact Explorer application tree and build manifest. A change to application bytes requires fresh runtime evidence, even when the frozen Heritage source corpus is unchanged. A prior receipt cannot establish that the new application was tested. Its earlier version remains available in Git history.

Refresh the 100-question suite and the three local journeys against the current assembled Site, preserving the original source snapshot dates. Serve the Site at a loopback root with /publication/ mapped internally to the repository's checked evaluation/heritage/ corpus directory. The Site intentionally excludes those runtime corpus files. Use --bundle for the question suite and --candidate-bundle plus --bundle-root for the journeys, so tiny, faithful and synthetic fixtures keep their own starting bundles. Materialise the new results with scripts/build_heritage_local_candidate_receipt.py, supplying both fresh result files, the current Site candidate receipt and an observation time after both executions. The materialiser checks scores, journey outcomes, source roots, app identity and timestamps before writing evidence.

The reading-help corpus update preserved the preceding receipt and both compressed browser results byte for byte under the repository path validation/reading-help/2026-09-27/pre-corpus-consumer/. The visible-help update also preserves the preceding receipt and both compressed results byte for byte under validation/reading-help/2026-10-01/pre-visible-help/. Its refreshed 100-question suite and three local journeys bind the application that displays help on selection.

The current receipt is regenerated only from fresh runs against the assembled Site and its exact Explorer build. The 100 questions are deterministic browser checks, not model-authored legal answers.

The beginner entry-link update preserves the preceding receipt and both compressed results under validation/beginner-entry/2026-10-24/pre-entry-links/, with a SHA-256 inventory. Fresh runs bind its navigation changes to the current application build while retaining the original Heritage source snapshot.

With the Site and /publication/ alias served at http://127.0.0.1:8002/, the browser commands are:

node scripts/evaluate_okf_explorer.mjs \
  --base-url http://127.0.0.1:8002/ \
  --bundle /publication/okf-explorer.json \
  --suite evaluation-foundry/fixtures/heritage-warwickshire/questions.json \
  --limit 100 --out tmp/heritage-questions

node scripts/evaluate_okf_explorer.mjs \
  --base-url http://127.0.0.1:8002/ \
  --candidate-bundle /okf-explorer.json \
  --bundle-root http://127.0.0.1:8002/publication/ \
  --journeys-only \
  --journeys evaluation-foundry/fixtures/heritage-warwickshire/journeys.json \
  --journey-id journey-tiny --journey-id journey-faithful \
  --journey-id journey-synthetic-isolation --out tmp/heritage-journeys

Keep the journey candidate root-relative as shown: the receipt expects that declaration and verifies the resolved URL. Do not pass --bundle for the journey run; it deliberately overrides all starting bundles and would replace the tiny and synthetic fixtures with the faithful corpus. Set the documented Playwright package and browser executable environment variables when using a local browser.

These are local Explorer runtime checks. Public deployment verification and domain-specific policy evaluation remain separately recorded activities.

Fresh candidate observations from CI

The repository owner maintains the separate heritage-candidate-evidence CI job. It runs when an app or Site change requires fresh observations and can also run through the existing manual CI workflow. It uses that run's exact app artefact, assembles the Site at the checked-out commit, and executes the same 100 questions and three journeys above with installed Google Chrome. It has read-only repository permissions, a 20-minute deadline and seven-day artefact retention. It changes no deployment, workflow trigger or retained-receipt gate.

The bounded local server is scripts/serve_heritage_candidate.mjs. It binds only to 127.0.0.1, serves regular files within the assembled _site and checked Heritage corpus roots, rejects traversal and symlink escapes, and accepts only GET and HEAD. Site files take precedence; /publication/ explicitly addresses the corpus, with root corpus aliases retained for the canonical journey. It streams files without executing them and performs no remote proxying. The default port is 8002; HERITAGE_EVIDENCE_PORT accepts an integer from 1 to 65535.

The threats are stale app identity, changed source roots, failed source access, unsafe file disclosure and mistaking observations for acceptance. The ordinary app and adversarial gates precede execution. The job uses only the reviewed public Heritage fixtures and declared source links; it copies no account-specific payload or credentials. A failed navigation remains a failed journey. A local network restriction must not be avoided by changing an assertion or substituting a source response. Running the unchanged checks on an authorised CI runner is separate execution evidence, not a proxy around the local environment.

The artefact heritage-candidate-evidence-COMMIT contains only the existing okf-explorer-evaluation-results.v1 question and journey JSON/Markdown, the okf-site-candidate-receipt.v1 Site receipt, the app build manifest, publication identity and a plain source-commit.txt. COMMIT is the checked-out workflow commit, including the PR merge candidate where applicable. The producer never writes the tracked local receipt, commits changes or promotes a deployment. The independent python-contracts validation remains required.

When an authorised review environment cannot download artefact storage, the same successful job emits a bounded text transport in its ordinary log. The helper scripts/emit_heritage_candidate_transport.py reads only six fixed public files: the two result JSON files, Site receipt, app manifest, publication identity and source commit. It rejects symlinks, non-regular files, files over 1 MiB, total input over 2 MiB, ZIP output over 256 KiB and log output over 384 KiB. It makes no network request and reads no environment values. The deterministic ZIP uses fixed timestamps, order and permissions. OKF_HERITAGE_PUBLIC_EVIDENCE_V1 BEGIN declares ZIP byte length, SHA-256 and chunk count; numbered CHUNK n/total lines contain at most 8192 base64 characters, followed by an END marker. The complete payload is validated before any part is printed. Normal artefact upload remains. For log transport, require one complete envelope from the successful producer job, all chunks exactly once in order, strict base64 decoding, the declared byte length and SHA-256, and exactly the six regular ZIP entries before extraction. This transport changes no result schema or acceptance requirement.

To import an observation, first verify the workflow run, source commit and downloaded artefact identity against the reviewed candidate. Compare its exact app tree and manifest with the local checked build. Preserve the previous receipt and both compressed results byte for byte, then run the existing materialiser with the downloaded inputs:

uv run --locked python scripts/build_heritage_local_candidate_receipt.py \
  --question-results tmp/heritage-candidate/questions/results.json \
  --journey-results tmp/heritage-candidate/journeys/results.json \
  --site-candidate-receipt tmp/heritage-candidate/site-candidate-receipt.json \
  --observed-at OBSERVATION_TIME_AFTER_BOTH_RUNS

The materialiser independently validates scores, journey outcomes, corpus roots, app identity, timestamps and result hashes. Review and commit its three generated files separately; rerun the retained-receipt tests. Failed or unmatched results cannot supply acceptance. The receipt records the executed Site candidate. A later evidence-only commit changes the publication identity of a newly assembled Site; it does not retroactively change the recorded candidate or establish a deployed merge commit. Pages has its own exact-commit verification.

The compact facet update's retained observations came from CI run 40105915998, at checked-out source 5021656698ebc424013e976c178cfc89e49268ca. All 100 questions passed with a mean score of 92.6; all three journeys passed their 40 actions and 26 assertions. The imported receipt retains that executed Site identity and binds the compact app tree d127aadb96a77711d5f7cc7aec8cf731cbf5ec1e3d49304d8132189ea4991f93. The preceding receipt and both compressed results are preserved byte for byte under validation/compact-facets/2026-10-24/pre-compact/.

On 27 October 2026, the record source reference update received fresh observations from CI run 40492019206, successful job 122149016399. The reviewed PR head was f71f776253d54a51552f061a1b10b0b922b99023; the job executed merge candidate e3c0eaa1bd48b99320e57627639e5a1139867146. All 100 questions scored at least 80, with a mean of 92.6. The tiny, faithful and synthetic-isolation journeys all passed. Both result files and the Site receipt bind app tree 827ac61a364500a2c73ad949dfda36f0839e54056644edd414ddb34f96aee960 and build manifest bbb482b252b9385149ab06a0bb5cbaae778e65a9498ea5c06d3683f4aa297d7a. The retained evidence inventory links the previous receipt, unchanged CI observations and their SHA-256 inventories. The materialised receipt records this executed candidate; deployment verification remains a separate check.

The rubric is aligned with GOV.UK service-quality guidance: user needs, simple task completion, accessibility, open standards, security/privacy, measurable performance and reliable operation. Accessibility expectations use WCAG 2.2 AA as the public-service baseline and combine automated checks with manual visual review.

Assets

Graph layering and overlapping labels

The review note for that image is retained verbatim in the manifest: "See how messy this display is due to layering and overlapping white boxes and the arrow location (should be up to the start of the icon)".

Additional evidence now captures the OS Data Hub graph failures:

Interaction checks

The browser harness and static lockstep tests should treat these behaviours as publication requirements for large OKF packs:

The interaction manifests make these requirements executable. Their action vocabulary covers search, facet opening and selection, sorting, a browser Back/Forward round trip, graph-edge selection, pointer-drag resizing of the relationship drawer, full-record loading, disclosure folding, in-Explorer Source Inspector loading, Map reduction/record selection and opening the raw source in a separate tab. Their assertions check URL parameters, restored control values, edge selection, drawer size, disclosure states, Source Inspector presence and preservation of the Explorer tab.

The 100 retrieval questions remain backward compatible. Journey execution is opt-in with --journeys; a normal suite invocation still runs the same 100 query records and uses the same 100-point rubric.

Rubric

The score is additive and totals 100 points:

This is not an assurance score for the source APIs. It scores how well the OKF Explorer helps a reader inspect the available public metadata.

Running locally

Run the focused Map UI suite directly from the Explorer application:

cd apps/okf-explorer
pnpm test:e2e

After main has deployed, run the same 18 scenarios through the public Pages application. The deterministic bundle and preview fixtures remain intercepted; only the Explorer HTML, JavaScript and CSS come from the deployed URL:

cd apps/okf-explorer
pnpm test:e2e:pages

Playwright starts the local Vite server and intercepts deterministic public- origin fixtures. Geometry bodies are not requested before the preview action, and no live ArcGIS/OGC service is required. CI runs this suite after the Svelte unit tests and production build. Failure screenshots, traces and reports stay under ignored application output folders.

Journey validation cross-checks every playwright_test_ids value against the spec and fails if an implemented GEO-E2E-* scenario has no persona/story, or if documentation refers to a test that does not exist.

Build the static site first:

uv run --locked python scripts/build_site.py

Serve the repository root:

uv run --locked python -m http.server 8002 --bind 127.0.0.1

Run the full 100-question browser suite:

node scripts/evaluate_okf_explorer.mjs \
  --base-url http://127.0.0.1:8002/next/ \
  --bundle /uk-government-apis/okf-explorer.json \
  --limit 100 \
  --out evaluation/okf-explorer/results/latest

For a post-deployment review, use absolute Pages URLs so the project subpath is not mistaken for the GitHub account root:

PLAYWRIGHT_PACKAGE="$PWD/apps/okf-explorer/node_modules/@playwright/test" \
  node scripts/evaluate_okf_explorer.mjs \
  --base-url https://chris-page-gov.github.io/okf-explorer/ \
  --bundle https://chris-page-gov.github.io/okf-explorer/uk-government-apis/okf-explorer.json \
  --limit 100 \
  --journeys evaluation/okf-explorer/journeys.json

The journey contract treats the default relevance sort as canonical when the sort query parameter is absent. It validates both that omission and the restored sort control value after browser Back/Forward.

If Playwright is installed outside the repo, point the script at that module:

PLAYWRIGHT_PACKAGE=/Users/crpage/tmp/okf-playwright/node_modules/playwright \
  node scripts/evaluate_okf_explorer.mjs \
  --base-url http://127.0.0.1:8002/next/ \
  --bundle ../uk-government-apis/okf-explorer.json \
  --limit 100

Generated reports are written to evaluation/okf-explorer/results/, which is ignored by Git. The committed suite, rubric, screenshot evidence and harness are the source of truth.

For CI or a quick lockstep check, use --no-browser. That mode validates the 100-question suite and writes one validation-only record per question, but it does not assign retrieval/display/accessibility/GOV.UK scores because no browser observations have been collected.

Validate persona/story/question traceability and the interaction schema without launching a browser:

node scripts/evaluate_okf_explorer.mjs \
  --no-browser \
  --journeys-only \
  --journeys evaluation/okf-explorer/journeys.json

Run only the UK Government APIs interaction journeys:

node scripts/evaluate_okf_explorer.mjs \
  --base-url http://127.0.0.1:8002/next/ \
  --journeys-only \
  --journeys evaluation/okf-explorer/journeys.json

Run the GOV.UK CKAN parity suite against the hosted CKAN descriptor:

node scripts/evaluate_okf_explorer.mjs \
  --base-url http://127.0.0.1:8002/next/ \
  --suite evaluation/gov-ckan/questions.json \
  --limit 100

The CKAN suite declares its target_bundle; the harness also picks up the sibling visual-regressions.json, so --bundle and --visual are optional for that run. Reports are written to evaluation/gov-ckan/results/ unless --out is specified.

Run the three CKAN interaction journeys, including the source-data/new-tab check:

node scripts/evaluate_okf_explorer.mjs \
  --base-url http://127.0.0.1:8002/next/ \
  --journeys-only \
  --journeys evaluation/gov-ckan/journeys.json

The legislation journey manifest can be validated or run the same way by substituting evaluation/legislation/journeys.json. Its legal-answer questions continue to be scored by scripts/evaluate_legislation_answers.py; the browser journeys test the Explorer discovery stage, not legal correctness.

External bundle runtime evidence

An external producer can run a declarative journey manifest against its frozen bundle with:

pnpm acceptance:bundle -- \
  --bundle-root /absolute/path/to/bundle \
  --journeys /absolute/path/to/journeys.json \
  --bundle-label bundle \
  --journey-label evaluation/journeys.json \
  --output /absolute/path/to/runtime-receipt.json

The command acquires one checkout-scoped single-writer lock before it performs the deterministic Explorer build. A concurrent invocation is refused before it can alter the build directory, launch Chromium or create a receipt. The inner runner verifies the live lock, its wrapper purpose and its parent process before reading an input. On POSIX systems the wrapper retains the process-group identity, terminates descendants and releases the lock only after it verifies that the group has gone. A left-over lock is not silently broken: inspect it and confirm that no acceptance process is active before removing it. After the deterministic build succeeds, the wrapper durably adds the exact command, build script and canonical build-manifest identity to the live lock. The inner runner recomputes that attestation before Chromium starts and again after it closes; a merely present or previously generated build/ directory cannot satisfy the gate.

Journey actions may use a bounded capture_attributes step. For ranked search evidence, first wait for the expected canonical result and then capture the stable canonical URL attributes exposed by Explorer:

[
  {
    "type": "wait_for_ranked_result",
    "canonical_url": "https://www.gov.uk/government/organisations/land-registry"
  },
  {
    "type": "capture_attributes",
    "id": "ranked-result-urls",
    "selector": "[data-okf-ranked-results=\"primary\"] [data-okf-ranked-result]",
    "name": "data-result-canonical-url",
    "min_items": 1,
    "max_items": 10
  }
]

wait_for_ranked_result derives the expected query from the current non-empty q URL parameter. It waits until the primary result list reports both that exact query and data-okf-search-state="settled", then requires exactly one visible row with the declared credential-free absolute HTTP(S) canonical URL. A genuinely loading search retains the 90-second journey ceiling. A settled empty or wrong result fails immediately instead of waiting for an absent title selector. The same exact object can be used as a terminal assertion by changing its type to ranked_result.

The capture then reads count, order and attributes in one browser evaluation so a rerender cannot mix two result states. Every retained ranked row in the bounded capture must provide a non-empty, credential-free absolute HTTP(S) canonical URL; an absent or unsafe URL fails the journey instead of shifting later ranks.

The receipt retains the values in rendered order, together with the selector, attribute, bound and total number of matching elements. The manifest contract bounds bytes, journeys, actions, assertions, observations, selectors and text; runtime retention also bounds requests, console events, page errors and final receipt bytes. It accepts only href or data-* captures, a maximum of 100 values and unique observation identifiers within each journey. Explorer stages the bundle and app build into private independent snapshots, serves only those declared bytes through no-follow file descriptors, and rejects a source tree, private snapshot, journey manifest or executable material whose final identity no longer matches its initial value. Global entry, file, byte, telemetry, journey-time and run-time limits are enforced before retained evidence is published. Request assertions use only the current journey and are evaluated after its browser context closes, so a late request, console error or page error cannot evade the terminal result. Retained browser-request URLs reject credentials and omit query strings and fragments.

The final receipt is written as a complete same-directory temporary file, flushed, linked into its previously absent destination without replacement, and followed by a directory flush and independent byte verification. The output parent is physically resolved outside both input trees and its directory identity is rechecked before and after publication. A producer's evaluator should bind the receipt to the exact bundle tree, manifest and Explorer consumer, then derive rank-sensitive metrics from this observation. A separate search model may remain diagnostic, but must not substitute for the product observation in a product acceptance gate.

Corpus Boundary Note

Question Q071 checks the user's Rugby search concern. In the current UK Government APIs OKF bundle, Rugby has one indexed match: Scarborough Borough Council New Local Plan Former Rugby Club Site. If future harvests add more Rugby records, the expected minimum can be raised without changing the harness.

The CKAN suite includes its own Rugby and planning questions so the broader CKAN fixture can be evaluated without assuming the same corpus boundary as the UK Government APIs pack.