OKF Explorer OKF Explorer documentation

OKF Explorer Evaluation Harness

This repository now carries repeatable browser evaluation suites for the UK Government APIs OKF pack, the hosted GOV.UK CKAN OKF pack, and future large OKF bundles.

The suite evaluates three publication goals:

The rubric is aligned with GOV.UK service-quality guidance: user needs, simple task completion, accessibility, open standards, security/privacy, measurable performance and reliable operation. Accessibility expectations use WCAG 2.2 AA as the public-service baseline and combine automated checks with manual visual review.

Assets

Graph layering and overlapping labels

The review note for that image is retained verbatim in the manifest: "See how messy this display is due to layering and overlapping white boxes and the arrow location (should be up to the start of the icon)".

Additional evidence now captures the OS Data Hub graph failures:

Interaction Checks

The browser harness and static lockstep tests should treat these behaviours as publication requirements for large OKF packs:

The interaction manifests make these requirements executable. Their action vocabulary covers search, facet opening and selection, sorting, a browser Back/Forward round trip, graph-edge selection, pointer-drag resizing of the relationship drawer, full-record loading, disclosure folding, in-Explorer Source Inspector loading, Map reduction/record selection and opening the raw source in a separate tab. Their assertions check URL parameters, restored control values, edge selection, drawer size, disclosure states, Source Inspector presence and preservation of the Explorer tab.

The 100 retrieval questions remain backward compatible. Journey execution is opt-in with --journeys; a normal suite invocation still runs the same 100 query records and uses the same 100-point rubric.

Rubric

The score is additive and totals 100 points:

This is not an assurance score for the source APIs. It scores how well the OKF Explorer helps a reader inspect the available public metadata.

Running Locally

Run the focused Map UI suite directly from the Explorer application:

cd apps/okf-explorer
pnpm test:e2e

After main has deployed, run the same 18 scenarios through the public Pages application. The deterministic bundle and preview fixtures remain intercepted; only the Explorer HTML, JavaScript and CSS come from the deployed URL:

cd apps/okf-explorer
pnpm test:e2e:pages

Playwright starts the local Vite server and intercepts deterministic public- origin fixtures. Geometry bodies are not requested before the preview action, and no live ArcGIS/OGC service is required. CI runs this suite after the Svelte unit tests and production build. Failure screenshots, traces and reports stay under ignored application output folders.

Journey validation cross-checks every playwright_test_ids value against the spec and fails if an implemented GEO-E2E-* scenario has no persona/story, or if documentation refers to a test that does not exist.

Build the static site first:

uv run --locked python scripts/build_site.py

Serve the repository root:

uv run --locked python -m http.server 8002 --bind 127.0.0.1

Run the full 100-question browser suite:

node scripts/evaluate_okf_explorer.mjs \
  --base-url http://127.0.0.1:8002/next/ \
  --bundle /uk-government-apis/okf-explorer.json \
  --limit 100 \
  --out evaluation/okf-explorer/results/latest

For a post-deployment review, use absolute Pages URLs so the project subpath is not mistaken for the GitHub account root:

PLAYWRIGHT_PACKAGE="$PWD/apps/okf-explorer/node_modules/@playwright/test" \
  node scripts/evaluate_okf_explorer.mjs \
  --base-url https://chris-page-gov.github.io/okf-explorer/ \
  --bundle https://chris-page-gov.github.io/okf-explorer/uk-government-apis/okf-explorer.json \
  --limit 100 \
  --journeys evaluation/okf-explorer/journeys.json

The journey contract treats the default relevance sort as canonical when the sort query parameter is absent. It validates both that omission and the restored sort control value after browser Back/Forward.

If Playwright is installed outside the repo, point the script at that module:

PLAYWRIGHT_PACKAGE=/Users/crpage/tmp/okf-playwright/node_modules/playwright \
  node scripts/evaluate_okf_explorer.mjs \
  --base-url http://127.0.0.1:8002/next/ \
  --bundle ../uk-government-apis/okf-explorer.json \
  --limit 100

Generated reports are written to evaluation/okf-explorer/results/, which is ignored by Git. The committed suite, rubric, screenshot evidence and harness are the source of truth.

For CI or a quick lockstep check, use --no-browser. That mode validates the 100-question suite and writes one validation-only record per question, but it does not assign retrieval/display/accessibility/GOV.UK scores because no browser observations have been collected.

Validate persona/story/question traceability and the interaction schema without launching a browser:

node scripts/evaluate_okf_explorer.mjs \
  --no-browser \
  --journeys-only \
  --journeys evaluation/okf-explorer/journeys.json

Run only the UK Government APIs interaction journeys:

node scripts/evaluate_okf_explorer.mjs \
  --base-url http://127.0.0.1:8002/next/ \
  --journeys-only \
  --journeys evaluation/okf-explorer/journeys.json

Run the GOV.UK CKAN parity suite against the hosted CKAN descriptor:

node scripts/evaluate_okf_explorer.mjs \
  --base-url http://127.0.0.1:8002/next/ \
  --suite evaluation/gov-ckan/questions.json \
  --limit 100

The CKAN suite declares its target_bundle; the harness also picks up the sibling visual-regressions.json, so --bundle and --visual are optional for that run. Reports are written to evaluation/gov-ckan/results/ unless --out is specified.

Run the three CKAN interaction journeys, including the source-data/new-tab check:

node scripts/evaluate_okf_explorer.mjs \
  --base-url http://127.0.0.1:8002/next/ \
  --journeys-only \
  --journeys evaluation/gov-ckan/journeys.json

The legislation journey manifest can be validated or run the same way by substituting evaluation/legislation/journeys.json. Its legal-answer questions continue to be scored by scripts/evaluate_legislation_answers.py; the browser journeys test the Explorer discovery stage, not legal correctness.

External bundle runtime evidence

An external producer can run a declarative journey manifest against its frozen bundle with:

pnpm acceptance:bundle -- \
  --bundle-root /absolute/path/to/bundle \
  --journeys /absolute/path/to/journeys.json \
  --bundle-label bundle \
  --journey-label evaluation/journeys.json \
  --output /absolute/path/to/runtime-receipt.json

The command acquires one checkout-scoped single-writer lock before it performs the deterministic Explorer build. A concurrent invocation is refused before it can alter the build directory, launch Chromium or create a receipt. The inner runner verifies the live lock, its wrapper purpose and its parent process before reading an input. On POSIX systems the wrapper retains the process-group identity, terminates descendants and releases the lock only after it verifies that the group has gone. A left-over lock is not silently broken: inspect it and confirm that no acceptance process is active before removing it. After the deterministic build succeeds, the wrapper durably adds the exact command, build script and canonical build-manifest identity to the live lock. The inner runner recomputes that attestation before Chromium starts and again after it closes; a merely present or previously generated build/ directory cannot satisfy the gate.

Journey actions may use a bounded capture_attributes step. For ranked search evidence, first wait for the expected canonical result and then capture the stable canonical URL attributes exposed by Explorer:

[
  {
    "type": "wait_for_ranked_result",
    "canonical_url": "https://www.gov.uk/government/organisations/land-registry"
  },
  {
    "type": "capture_attributes",
    "id": "ranked-result-urls",
    "selector": "[data-okf-ranked-results=\"primary\"] [data-okf-ranked-result]",
    "name": "data-result-canonical-url",
    "min_items": 1,
    "max_items": 10
  }
]

wait_for_ranked_result derives the expected query from the current non-empty q URL parameter. It waits until the primary result list reports both that exact query and data-okf-search-state="settled", then requires exactly one visible row with the declared credential-free absolute HTTP(S) canonical URL. A genuinely loading search retains the 90-second journey ceiling. A settled empty or wrong result fails immediately instead of waiting for an absent title selector. The same exact object can be used as a terminal assertion by changing its type to ranked_result.

The capture then reads count, order and attributes in one browser evaluation so a rerender cannot mix two result states. Every retained ranked row in the bounded capture must provide a non-empty, credential-free absolute HTTP(S) canonical URL; an absent or unsafe URL fails the journey instead of shifting later ranks.

The receipt retains the values in rendered order, together with the selector, attribute, bound and total number of matching elements. The manifest contract bounds bytes, journeys, actions, assertions, observations, selectors and text; runtime retention also bounds requests, console events, page errors and final receipt bytes. It accepts only href or data-* captures, a maximum of 100 values and unique observation identifiers within each journey. Explorer stages the bundle and app build into private independent snapshots, serves only those declared bytes through no-follow file descriptors, and rejects a source tree, private snapshot, journey manifest or executable material whose final identity no longer matches its initial value. Global entry, file, byte, telemetry, journey-time and run-time limits are enforced before retained evidence is published. Request assertions use only the current journey and are evaluated after its browser context closes, so a late request, console error or page error cannot evade the terminal result. Retained browser-request URLs reject credentials and omit query strings and fragments.

The final receipt is written as a complete same-directory temporary file, flushed, linked into its previously absent destination without replacement, and followed by a directory flush and independent byte verification. The output parent is physically resolved outside both input trees and its directory identity is rechecked before and after publication. A producer's evaluator should bind the receipt to the exact bundle tree, manifest and Explorer consumer, then derive rank-sensitive metrics from this observation. A separate search model may remain diagnostic, but must not substitute for the product observation in a product acceptance gate.

Corpus Boundary Note

Question Q071 checks the user's Rugby search concern. In the current UK Government APIs OKF bundle, Rugby has one indexed match: Scarborough Borough Council New Local Plan Former Rugby Club Site. If future harvests add more Rugby records, the expected minimum can be raised without changing the harness.

The CKAN suite includes its own Rugby and planning questions so the broader CKAN fixture can be evaluated without assuming the same corpus boundary as the UK Government APIs pack.