Code Review Follow-up — ai-infrastructure-wiki

Date: 2026-07-08 Scope: review of docs/code-review-2026-07-07.md, the local fix-session commits described in that document, and the remaining work required to reach a clean, publishable state.

Executive Summary

docs/code-review-2026-07-07.md is useful and mostly accurate as a combined review report plus implementation log. It correctly identifies the most important issue: the published UK Government APIs corpus still contains inherited credential-like query parameters even though the aims document forbids stored secret values. It also correctly records that Claude Fable/Sonnet implemented generator-side and Explorer-side fixes for most of the code defects.

The important correction is that the fix session was not complete from a publication perspective. The code had been fixed locally, but the generated uk-government-apis/ corpus had not been regenerated with the fixed generator. Therefore the repository still contained the credential-bearing URLs in the committed generated data, and the public Pages deployment from origin/main was still stale. Completion had to be treated as a gated publication task, not a documentation cleanup.

Current status after merge: this follow-up is now a historical completion report plus current-state checkpoint. The completion branch was merged, and later Explorer work on main added facet/search/graph/timeline improvements and throttled large-corpus shard hydration to reduce transient GitHub Pages 503s. Current local main is synced to origin/main at ae7879bd (Throttle large-corpus shard hydration).

Review-time local state:

• At the start of this follow-up, local main was 12 commits ahead of origin/main and the generated corpus had not yet been republished.
• That review-time state is no longer current. The remediation and subsequent Explorer fixes are now merged to main.
• The current repository should be checked with git status --short --branch before acting on any dated branch-status statement below.

Implementation update:

• The branch codex/complete-code-review-fixes now carries this completion work.
• The live UK Government APIs corpus has been regenerated with redaction, safety checks, derived records and the wider exemplar improvements.
• Current generated counts are 244 declared API products, 13 provider-native API products, 34,327 data access endpoints, 7,827 data products, 2,994 contracts/capability documents, 38 operations, 93 schemas, 274 publishers and 302,683 relationships.
• A structured scan of generated JSON found zero credential-like URL query parameters across 360,584 generated URLs.
• The generator now emits Markdown OKF files for selected record classes: 3,383 api-records/*.md, 274 organisations/*.md, plus index.md and log.md.

Review of the 2026-07-07 Report

What It Gets Right

The report correctly separates three layers that are easy to conflate:

1. Explorer security defects in Svelte, the static explorer, and the legacy viewer.
2. Generator defects around URL safety, redaction, deterministic harvests, text sanitisation, and relationship provenance.
3. Exemplar completeness gaps around Markdown OKF output, declared-product crosslinks, contract discovery, organisation reconciliation, samples, and credential modelling.

It also correctly classifies the inherited credential URLs as a real no-secrets violation, not merely an agent false positive. At review time, a fresh scan of the then-current generated corpus found:

unique_urls_with_credential_params: 54
param_occurrences:
  access_token: 4
  login: 192
  password: 192
  token: 20

Those occurrence counts are inflated by repeated representations in generated shards, but the underlying issue is real. The exact numbers vary depending on whether the scan counts unique URLs, fields, generated shards, or individual query parameters.

The report also correctly records that fixture generation with the fixed generator redacts credentials and surfaces counters. A local fixture run produced no password=, token=, access_token=, or login= URL matches and reported:

{
  "credential_parameters_redacted": 3,
  "duplicate_endpoints_skipped": 1,
  "duplicate_slugs_dropped": 0,
  "unsafe_urls_dropped": 1
}

What Was Stale or Too Optimistic

The report's section 7 says every Open finding except the component-test gap has been fixed. At review time, that was accurate only for code paths and fixture output. It was not yet true for the checked-in generated corpus or the public URL.

The report marks R1 as "minutes + review." In practice R1 was the highest-risk gate because it was the step that removed credential-bearing URLs from the committed and published bundle. It had to be promoted from a remaining task to a release blocker.

The report also says the worktree was clean and build_site.py passed at close. In this local environment, the repository had ignored .DS_Store files in paths copied into _site, so build_site.py failed until those files were removed. This was not a code regression, but it was a real local pre-flight requirement.

Finally, the report records the fix commits as local and unpushed. That remained true relative to origin/main; the completion plan therefore needed a branch/PR from the local fixed state rather than assuming the fixes were already on GitHub.

Pre-implementation Verification Snapshot

Commands run on 2026-07-08:

The CI workflow added by the fix session is directionally correct: it pins Python 3.12, runs Svelte validation, runs Python unit tests, exercises fixture-based UK API generation into $RUNNER_TEMP, and builds the static site. GitHub CI should not see local .DS_Store files, but the local pre-flight should clean them so the developer check suite matches CI.

Completion Plan to Pass All Checks

Phase 0 — Stabilise the Branch

Create a real feature branch from the local fixed main state before making any more changes:

git checkout -B codex/complete-code-review-fixes

Do not push local main directly. The branch should contain the 12 existing local fix commits plus the final corpus regeneration commit and this report.

Phase 1 — Clean Local Build Hygiene

Remove ignored Finder metadata before running publication checks:

find . -name .DS_Store -delete

This is required because scripts/build_site.py intentionally rejects forbidden files after copying public trees into _site/. The files are ignored by Git, but they still affect the local site build.

Phase 2 — Regenerate the UK Government APIs Corpus

Run the live generator with the fixed redaction and provenance logic:

python3 scripts/build_uk_government_api_okf.py

This is the release-blocking step. It should update uk-government-apis/ so that:

• credential-bearing query parameters are removed from persisted URLs;
• data-hygiene warnings appear in overview/analysis outputs;
• unsafe harvested URLs are dropped;
• text sanitisation changes are reflected in generated records;
• generated relationships include edge provenance/confidence where supported by the new generator;
• python3 scripts/build_uk_government_api_okf.py --check passes afterward against live sources.

After regeneration, run a credential scan over uk-government-apis/ and require zero matches for persisted URL query parameters named password, login, token, access_token, api_key, apikey, client_secret, secret, sig, signature, or keys ending in key.

Phase 3 — Run the Full Local Gate

Run the checks in this order:

python3 -m unittest discover -s tests
python3 scripts/build_uk_government_api_okf.py --check
python3 scripts/build_okf_bundle.py --check
python3 scripts/update_viewer.py --check
python3 scripts/check_okf.py
cd apps/okf-explorer && pnpm check && pnpm test && pnpm build && cd ../..
python3 scripts/build_site.py

Expected outcome:

• all checks pass;
• build_site.py reports a completed _site build;
• no forbidden files are copied;
• no credential-bearing URLs remain in generated corpus JSON;
• git status shows only intended tracked changes plus known untracked local screenshots if left outside the commit.

Phase 4 — Commit and PR

Commit the completion as a new commit on codex/complete-code-review-fixes. The commit should include:

• regenerated uk-government-apis/ data;
• this docs/code-review-2026-07-08.md report;
• any necessary small correction if the regeneration reveals a deterministic check failure.

Push and open a PR. The PR description should explicitly state that it closes the security-publication gap by regenerating the corpus with credential redaction.

Required GitHub checks:

• okf-explorer-ci passes on the PR;
• review requirement satisfied;
• admin override should not be needed unless the policy state requires it.

Phase 5 — Merge and Verify Publication

After merge, verify the Pages deployment and the stable URLs:

curl -fsSL https://chris-page-gov.github.io/okf-uk-government-apis/okf-explorer.json
curl -sI "https://chris-page-gov.github.io/okf-explorer/?bundle=https%3A%2F%2Fchris-page-gov.github.io%2Fai-infrastructure-wiki%2Fuk-government-apis%2Fokf-explorer.json&view=reader#overview"

Then re-run the credential scan against the published descriptor/shards if practical, or at minimum against the merged repository state.

Exemplar-improvement Work Completed With This Completion

The 2026-07-07 report listed several larger exemplar-improvement tasks as valuable but not required for the immediate security/publication completion. This pass completed them as part of the same branch so the exemplar is materially closer to the specification rather than merely cleared of credential-bearing URLs.

• Markdown OKF files are emitted for selected record classes, with endpoint-scale records retained in JSON shards to avoid turning the exemplar into tens of thousands of Markdown files.
• Declared API products are crosslinked to CKAN/OS/ONS records where a canonical provider and endpoint host can be matched; inferred links carry evidence type and confidence rather than pretending to be authoritative declarations.
• Contract and capability-document records are derived from observed OpenAPI, WSDL, OGC, ArcGIS, SPARQL and related metadata signals, and linked back to described records.
• Organisations now carry canonical publisher identifiers/titles for the provider set currently harvested, with alias handling for known GOV.UK/provider-name variants.
• The Explorer vocabulary now includes quality_band, relationship_density, assurance_status, canonical_publisher, data_classification and environment facets.
• Records include credential_requirements and static sample_policy fields, using references and policy descriptors only.
• The frontend test suite now includes a shipped-static-Explorer regression harness, and the Svelte source fetch helpers have byte-counted response tests.
• Svelte, worker and vanilla static Explorer JSON fetches now enforce streamed byte-count caps, not only Content-Length.

Remaining enrichment work should now be treated as iterative product depth rather than completion gating: deeper live contract parsing, fuller GOV.UK organisation-register reconciliation, richer provider-portal assurance and higher-fidelity operation/sample extraction.

Recommended Done Criteria

Completion is done only when all of the following are true:

1. The fixed generator has regenerated uk-government-apis/.
2. The generated corpus contains zero persisted credential-bearing URL query parameters.
3. The generated warnings counters record redactions/drops from live data.
4. The full local gate in Phase 3 passes.
5. The PR CI gate passes.
6. The PR is merged.
7. GitHub Pages deploys the merged commit.
8. The published descriptor is reachable at the stable URL.

Current branch status after this implementation pass:

Current Pack Checkpoint

Current generated descriptor state on main after the follow-up and later UI work:

• 257 API products, including 244 GOV.UK-declared products and 13 provider-native API products.
• 34,327 data access endpoints and 7,827 data products.
• 2,994 contracts/capability documents, 38 operations and 93 schemas.
• 274 providers/publishers and 302,683 relationships.
• 3,383 selected api-records/*.md files, 274 organisations/*.md files, plus uk-government-apis/index.md and uk-government-apis/log.md.
• All scanned relationships carry confidence, evidence_type and observed_at provenance fields.
• Generated warnings record 102 credential-parameter redactions, 5,892 duplicate endpoints skipped, 5 duplicate slugs dropped, 25,397 licences inferred from provider terms, 12,289 missing licences, 202 unknown access models and 17 missing contracts.
• A repository scan for persisted URL query parameters named password, login, token, access_token, api_key, apikey, client_secret, secret, sig or signature returns zero matches.

The 2026-07-07 report should therefore be read as the original problem record, not as the current state of the exemplar.