"@context":
  - https://chris-page-gov.github.io/okf-explorer/profile/bundle-wiki/v1/context.jsonld
  - https://chris-page-gov.github.io/okf-explorer/profile/bundle-wiki/v1/semantic-context.jsonld
"@id": https://chris-page-gov.github.io/okf-explorer/docs/postmortems/heritage-foundry-2026/methodology.html
"@type": https://schema.org/Report
type: Report
title: "Postmortem methodology and metric definitions"
description: "Scope, extraction, evidence, measurement and limitation rules for the heritage Foundry postmortem."
generated:
  by: process:heritage-foundry-postmortem-builder
  at: "2026-08-04T13:16:54Z"
assertion_status: normalized
assertion_scope: real-world
tags:
  - postmortem
  - methodology
  - evidence

Postmortem Methodology And Metric Definitions

Decision And Audience

The report is for maintainers and technically interested reviewers deciding how to change the Evaluation Foundry so late errors invalidate only their dependency cone. It is not an assessment of Historic England, GitHub or the end user's browsing behaviour.

Evidence Scope

The collection boundary is the task from 2026-08-02T21:46:49Z through the final publication-closure handoff, plus repository/GitHub evidence for PRs #67–#69. The private plane contains raw GitHub logs, structured PR/run metadata, release assets, Git outputs and three deployment archives. The public plane contains hashes, normalized registers, bounded excerpts and this analysis.

The primary performance reconstruction ends at the terminal 3 August release. The 4 August postmortem publication is reported separately as a controlled documentation-only invalidation exercise; it is not added to the three historical PR totals or six GitHub workflow totals. Current PR #70 and the replacement external publication are recorded through the normalized publication-evidence register; pending records do not change historical timing metrics or imply success.

The prior hackathon postmortem pattern was reused: one exchange begins with a visible user prompt and contains every visible assistant commentary/final message until the next prompt. System and developer instructions, private reasoning, tool arguments and tool outputs are not part of a prompt-response trace. Publication-evidence records are never converted into conversation messages, so the same rollout bytes always produce the same full trace regardless of rollout milestone status.

Metric Definitions

• Workflow wall time is updatedAt - startedAt for a GitHub Actions run. Times across runs are additive resource/queue observations, not elapsed delivery latency when work overlaps.
• CI job time is job completion minus job start. A step duration is computed from its GitHub job timestamps.
• File touch is one path reported by git diff --name-only for one PR phase. A path changed in two PRs counts twice in the total 4,280.
• Generated amplification is generated output touched because an upstream change invalidated broad output, compared with the number/type of substantive source changes. It is descriptive; no CPU cost per file is inferred.
• Late finding is a defect, inconsistency or release risk first reported after full-corpus generation or during Site/browser/publication assurance.
• Dependency cone is the changed input plus every transitively dependent producer plane, consumer test, Site component and promotion check.
• Outcome fingerprint is a stable projection of semantic results with run timestamps and observation-only metadata removed.
• Reused gate is a previously passed result whose declared inputs/roots are unchanged by the reviewed change class. The fresh Site-shell receipt names every reused and rerun gate; reuse is not inferred from a green final build.

Conversation Extraction

The curated source has SHA-256 0dbb30121104c82f4347c22aea32a0f670a25d8bfeae70cf8a4edd33cb66d6e6 and contained 58,524,942 bytes at extraction. The public trace contains 8 user exchanges and 309 visible Codex messages at extraction. Local paths and token-shaped strings are redacted. The public lint rejects local user paths, Codex rollout paths, private evidence paths and common token forms.

Command Evidence

The task runtime aggregates several nested commands into one orchestrator call. Therefore command invocation counts are exact for recognized executable command strings, while local per-command duration is unavailable when commands shared an outer call. GitHub job and step durations remain exact to the reported timestamp resolution. Raw command output stays private.

Limitations And Uncertainty

• This is one complex exemplar and three PRs, not a benchmark across projects.
• The trace cannot expose hidden model reasoning and does not claim to do so.
• Local tool calls made by subagents are represented through their visible summaries and repository results, not merged into the parent prompt trace.
• GitHub does not report billable cost here, so the report uses wall time and file/byte amplification rather than money.
• The release is digest-bound and frozen by policy, but GitHub reports isImmutable: false; this statement describes the historical 3 August release. The replacement policy is evaluated separately: the independently rooted 4 August R1 and R2 are counted as terminally passed only because the normalized evidence register supplies their exact immutable release identities, attestation URLs, asset digests and workflow runs.
• The current final response is included through the handoff capture mechanism; rerunning after the task completes can verify it against the finalized rollout.