# OKF Explorer Evaluation Harness

This repository now carries repeatable browser evaluation suites for the UK
Government APIs OKF pack, the hosted GOV.UK CKAN OKF pack, and future large OKF
bundles.

The suite evaluates three publication goals:

- Can a person browse efficiently, find an API or data source, and evaluate it?
- Does the display convey record, provenance, licence, access and quality
  information clearly?
- Are terms such as confidence, licence basis, API evidence and metadata quality
  defined in the UI instead of being unexplained percentages?

The rubric is aligned with GOV.UK service-quality guidance: user needs, simple
task completion, accessibility, open standards, security/privacy, measurable
performance and reliable operation. Accessibility expectations use WCAG 2.2 AA
as the public-service baseline and combine automated checks with manual visual
review.

## Assets

- `evaluation/okf-explorer/questions.json` contains 100 UK Government API
  retrieval and inspection tasks.
- `evaluation/okf-explorer/journeys.json` maps seven Explorer personas and
  nineteen broad and geospatial stories to those questions, defines browser
  interaction journeys and records the focused `GEO-E2E-*` Playwright tests
  attached to each Map story.
- `apps/okf-explorer/tests/ui/geospatial-map.spec.ts` contains 18 deterministic
  browser scenarios covering every Map control and visible state across small
  and large bundle paths.
- `evaluation/gov-ckan/questions.json` contains 100 GOV.UK CKAN retrieval and
  inspection tasks using the same rubric.
- `evaluation/gov-ckan/journeys.json` defines CKAN-specific personas and user
  stories, maps every CKAN question to at least one story, and covers durable
  retrieval state, graph interaction and source inspection.
- `evaluation/legislation/journeys.json` maps the six legislation personas and
  critical journeys to all 100 legal-answer questions. It records the curator
  refresh story as an explicit evaluation gap because that operation is covered
  by generator and documentation-lockstep tests rather than legal questions.
- `evaluation/okf-explorer/visual-regressions.json` records known visual
  clarity issues that must not be lost during redesign.
- `evaluation/gov-ckan/visual-regressions.json` intentionally remains empty
  until a CKAN screenshot is copied into the repository with its provenance;
  temporary or conversational screenshots are not silently turned into a
  committed visual baseline.
- `evaluation/okf-explorer/evidence/graph-layering-overlap-2026-07-08.png`
  captures the current graph readability problem:

![Graph layering and overlapping labels](../evaluation/okf-explorer/evidence/graph-layering-overlap-2026-07-08.png)

The review note for that image is retained verbatim in the manifest:
"See how messy this display is due to layering and overlapping white boxes and
the arrow location (should be up to the start of the icon)".

Additional evidence now captures the OS Data Hub graph failures:

- `graph-osdatahub-search-context-2026-07-08.png`: search context was not
  preserved clearly when moving from search results to Graph.
- `graph-osdatahub-cluster-overlap-2026-07-08.png`: zoom did not make a dense
  OS Data Hub cluster readable.
- `facet-record-type-graph-context-2026-07-08.png`: record-type counts looked
  like a false graph breakdown when the active reduction was not explicit.

## Interaction Checks

The browser harness and static lockstep tests should treat these behaviours as
publication requirements for large OKF packs:

- Closed facet sections must not render or scan their full value lists. Opening
  a facet should show a facet-local search box, loading state if hydration is
  needed, and paged values.
- Provider presentation defaults must affect facet labels, order, initial
  pinning, control choice and panel tabs. Device-local user pin/hide/reorder
  preferences take precedence for one bundle and **Reset** restores the
  provider profile.
- Manageable facets expose a proportional distribution with a textual summary
  only when the available vocabulary counts are complete; truncated top-value
  data is labelled as a preview. High-cardinality facets expose examples and
  search without rendering a long default list. A corpus with tens of thousands
  of values requires a sharded facet vocabulary rather than eager hydration.
- Facet search must normalise hyphenated, underscored and spaced terms the same
  way for the query and the candidate value.
- A plain click on a facet bar segment or value previews it without changing the
  filter URL; Ctrl-click or Cmd-click builds a highlighted set. Double-click,
  Enter or the explicit commit control applies that set as a filter.
- Low-cardinality facet distributions must be available before a facet is opened
  and must remain numerically consistent after full-record hydration. The ONS
  regression fixture expects 382 `region` records both before and after Graph
  loads nested `geography_metadata`.
- Facets support direct pinning, multiple open pinned sections and drag reorder,
  with action-menu move controls as the keyboard fallback. No legacy **Adjust**
  action is offered and value right-click has no selection meaning.
- A plain click on a graph node inspects it in the data card. A double-click is
  the graph navigation/reduction gesture.
- Dense graph stack expansion should re-group by a visible semantic dimension
  when expanding every record would make labels, arrows or cards unreadable.
- The graph legend must show the actual node-shape vocabulary, including stack
  and opened-stack group states.
- The relationship drawer must be independently scrollable and resizable
  without covering the primary graph target.
- Timeline views must be ordered by time, provide a Latest view, and offer
  grouped year, quarter and month buckets where dated metadata exists.
- Browser Back and Forward must preserve inspect/reduce context or expose a
  disabled state when no in-app forward target exists.

The interaction manifests make these requirements executable. Their action
vocabulary covers search, facet opening and selection, sorting, a browser
Back/Forward round trip, graph-edge selection, pointer-drag resizing of the
relationship drawer, full-record loading, disclosure folding, in-Explorer
Source Inspector loading, Map reduction/record selection and opening the raw
source in a separate tab. Their
assertions check URL parameters, restored control values, edge selection,
drawer size, disclosure states, Source Inspector presence and preservation of
the Explorer tab.

The 100 retrieval questions remain backward compatible. Journey execution is
opt-in with `--journeys`; a normal suite invocation still runs the same 100
query records and uses the same 100-point rubric.

## Rubric

The score is additive and totals 100 points:

- Retrieval, 35 points: result visibility, expected terms/routes, result count,
  stale-context reset, and core metadata discoverability.
- Display, 25 points: detail card title/summary/route, licence/access/contract
  visibility, clean metadata-gap wording, follow-on exploration controls,
  loading/search clarity, and graph readability.
- Accessibility, 20 points: accessible names, live status regions, keyboard
  focus targets, landmarks, and absence of obvious overlap at the tested
  viewport.
- GOV.UK-aligned publication quality, 20 points: plain language, provenance,
  licence/access clarity, predictable service-style actions, metadata-quality
  explanations, and no displayed secrets.

This is not an assurance score for the source APIs. It scores how well the OKF
Explorer helps a reader inspect the available public metadata.

## Running Locally

Run the focused Map UI suite directly from the Explorer application:

```sh
cd apps/okf-explorer
pnpm test:e2e
```

After `main` has deployed, run the same 18 scenarios through the public Pages
application. The deterministic bundle and preview fixtures remain intercepted;
only the Explorer HTML, JavaScript and CSS come from the deployed URL:

```sh
cd apps/okf-explorer
pnpm test:e2e:pages
```

Playwright starts the local Vite server and intercepts deterministic public-
origin fixtures. Geometry bodies are not requested before the preview action,
and no live ArcGIS/OGC service is required. CI runs this suite after the Svelte
unit tests and production build. Failure screenshots, traces and reports stay
under ignored application output folders.

Journey validation cross-checks every `playwright_test_ids` value against the
spec and fails if an implemented `GEO-E2E-*` scenario has no persona/story, or
if documentation refers to a test that does not exist.

Build the static site first:

```sh
uv run --locked python scripts/build_site.py
```

Serve the repository root:

```sh
uv run --locked python -m http.server 8002 --bind 127.0.0.1
```

Run the full 100-question browser suite:

```sh
node scripts/evaluate_okf_explorer.mjs \
  --base-url http://127.0.0.1:8002/next/ \
  --bundle /uk-government-apis/okf-explorer.json \
  --limit 100 \
  --out evaluation/okf-explorer/results/latest
```

For a post-deployment review, use absolute Pages URLs so the project subpath is
not mistaken for the GitHub account root:

```sh
PLAYWRIGHT_PACKAGE="$PWD/apps/okf-explorer/node_modules/@playwright/test" \
  node scripts/evaluate_okf_explorer.mjs \
  --base-url https://chris-page-gov.github.io/okf-explorer/ \
  --bundle https://chris-page-gov.github.io/okf-explorer/uk-government-apis/okf-explorer.json \
  --limit 100 \
  --journeys evaluation/okf-explorer/journeys.json
```

The journey contract treats the default `relevance` sort as canonical when the
`sort` query parameter is absent. It validates both that omission and the
restored sort control value after browser Back/Forward.

If Playwright is installed outside the repo, point the script at that module:

```sh
PLAYWRIGHT_PACKAGE=/Users/crpage/tmp/okf-playwright/node_modules/playwright \
  node scripts/evaluate_okf_explorer.mjs \
  --base-url http://127.0.0.1:8002/next/ \
  --bundle ../uk-government-apis/okf-explorer.json \
  --limit 100
```

Generated reports are written to `evaluation/okf-explorer/results/`, which is
ignored by Git. The committed suite, rubric, screenshot evidence and harness
are the source of truth.

For CI or a quick lockstep check, use `--no-browser`. That mode validates the
100-question suite and writes one validation-only record per question, but it
does not assign retrieval/display/accessibility/GOV.UK scores because no browser
observations have been collected.

Validate persona/story/question traceability and the interaction schema without
launching a browser:

```sh
node scripts/evaluate_okf_explorer.mjs \
  --no-browser \
  --journeys-only \
  --journeys evaluation/okf-explorer/journeys.json
```

Run only the UK Government APIs interaction journeys:

```sh
node scripts/evaluate_okf_explorer.mjs \
  --base-url http://127.0.0.1:8002/next/ \
  --journeys-only \
  --journeys evaluation/okf-explorer/journeys.json
```

Run the GOV.UK CKAN parity suite against the hosted CKAN descriptor:

```sh
node scripts/evaluate_okf_explorer.mjs \
  --base-url http://127.0.0.1:8002/next/ \
  --suite evaluation/gov-ckan/questions.json \
  --limit 100
```

The CKAN suite declares its `target_bundle`; the harness also picks up the
sibling `visual-regressions.json`, so `--bundle` and `--visual` are optional for
that run. Reports are written to `evaluation/gov-ckan/results/` unless `--out`
is specified.

Run the three CKAN interaction journeys, including the source-data/new-tab
check:

```sh
node scripts/evaluate_okf_explorer.mjs \
  --base-url http://127.0.0.1:8002/next/ \
  --journeys-only \
  --journeys evaluation/gov-ckan/journeys.json
```

The legislation journey manifest can be validated or run the same way by
substituting `evaluation/legislation/journeys.json`. Its legal-answer questions
continue to be scored by `scripts/evaluate_legislation_answers.py`; the browser
journeys test the Explorer discovery stage, not legal correctness.

## External bundle runtime evidence

An external producer can run a declarative journey manifest against its frozen
bundle with:

```sh
pnpm acceptance:bundle -- \
  --bundle-root /absolute/path/to/bundle \
  --journeys /absolute/path/to/journeys.json \
  --bundle-label bundle \
  --journey-label evaluation/journeys.json \
  --output /absolute/path/to/runtime-receipt.json
```

The command acquires one checkout-scoped single-writer lock before it performs
the deterministic Explorer build. A concurrent invocation is refused before it
can alter the build directory, launch Chromium or create a receipt. The inner
runner verifies the live lock, its wrapper purpose and its parent process before
reading an input. On POSIX systems the wrapper retains the process-group
identity, terminates descendants and releases the lock only after it verifies
that the group has gone. A left-over lock is not silently broken: inspect it
and confirm that no acceptance process is active before removing it.
After the deterministic build succeeds, the wrapper durably adds the exact
command, build script and canonical build-manifest identity to the live lock.
The inner runner recomputes that attestation before Chromium starts and again
after it closes; a merely present or previously generated `build/` directory
cannot satisfy the gate.

Journey actions may use a bounded `capture_attributes` step. For ranked search
evidence, first wait for the expected canonical result and then capture the
stable canonical URL attributes exposed by Explorer:

```json
[
  {
    "type": "wait_for_ranked_result",
    "canonical_url": "https://www.gov.uk/government/organisations/land-registry"
  },
  {
    "type": "capture_attributes",
    "id": "ranked-result-urls",
    "selector": "[data-okf-ranked-results=\"primary\"] [data-okf-ranked-result]",
    "name": "data-result-canonical-url",
    "min_items": 1,
    "max_items": 10
  }
]
```

`wait_for_ranked_result` derives the expected query from the current non-empty
`q` URL parameter. It waits until the primary result list reports both that
exact query and `data-okf-search-state="settled"`, then requires exactly one
visible row with the declared credential-free absolute HTTP(S) canonical URL.
A genuinely loading search retains the 90-second journey ceiling. A settled
empty or wrong result fails immediately instead of waiting for an absent title
selector. The same exact object can be used as a terminal assertion by changing
its type to `ranked_result`.

The capture then reads count, order and attributes in one browser evaluation
so a rerender cannot mix two result states. Every retained ranked row in the
bounded capture must provide a non-empty, credential-free absolute HTTP(S)
canonical URL; an absent or unsafe URL fails the journey instead of shifting
later ranks.

The receipt retains the values in rendered order, together with the selector,
attribute, bound and total number of matching elements. The manifest contract
bounds bytes, journeys, actions, assertions, observations, selectors and text;
runtime retention also bounds requests, console events, page errors and final
receipt bytes. It accepts only `href` or `data-*` captures, a maximum of 100
values and unique observation identifiers within each journey. Explorer stages
the bundle and app build into private independent snapshots, serves only those
declared bytes through no-follow file descriptors, and rejects a source tree,
private snapshot, journey manifest or executable material whose final identity
no longer matches its initial value. Global entry, file, byte, telemetry,
journey-time and run-time limits are enforced before retained evidence is
published. Request assertions use only the current journey and are evaluated
after its browser context closes, so a late request, console error or page error
cannot evade the terminal result. Retained browser-request URLs reject
credentials and omit query strings and fragments.

The final receipt is written as a complete same-directory temporary file,
flushed, linked into its previously absent destination without replacement,
and followed by a directory flush and independent byte verification. The
output parent is physically resolved outside both input trees and its directory
identity is rechecked before and after publication.
A producer's evaluator should bind the receipt to the exact bundle tree,
manifest and Explorer consumer, then derive rank-sensitive metrics from this
observation. A separate search model may remain diagnostic, but must not
substitute for the product observation in a product acceptance gate.

## Corpus Boundary Note

Question `Q071` checks the user's Rugby search concern. In the current UK
Government APIs OKF bundle, Rugby has one indexed match:
`Scarborough Borough Council New Local Plan Former Rugby Club Site`. If future
harvests add more Rugby records, the expected minimum can be raised without
changing the harness.

The CKAN suite includes its own Rugby and planning questions so the broader
CKAN fixture can be evaluated without assuming the same corpus boundary as the
UK Government APIs pack.