Stakeholder needs and evaluation journeys

Purpose and status

This is the evaluation design for the next stage of the independent DWP knowledge-framework exemplar. It adds five projected personas, six user stories and six proposed evaluation questions in knowledge/evaluation-journeys.yamlld. It extends the initial three personas, four stories and eight questions.

These are project-authored requirements and test designs. They have not been validated through user research or approved by policy, legal, operational or fraud and error specialists. No new evaluation outcomes are claimed here. The existing deterministic retrieval checks measure a narrower capability.

Staff needs below are de-identified paraphrases of owner-supplied correspondence. The correspondence, contact details and quotations are not published. A reference to an earlier Markdown plan appeared in the correspondence, but that plan was not supplied: its detailed requirements and acceptance criteria remain unknown.

What the contributions establish

The quoted model's technology choices and effort estimates are not adopted as staff-approved requirements. The corpus census and validation receipts in this repository are the evidence for work actually completed.

Projected personas and journeys

Each persona below is a hypothesis for research and evaluation, not a description of an identified person. The existing welfare rights adviser is reused for J04.

Proposed evaluation cases

Every case uses invented scenarios and no claimant personal data. A chapter or page listed below is a discovery starting point, not an assertion that it contains a complete answer or an operational requirement. The bundle's references edges are navigation links; they do not mean that a source legally authorises a rule.

J01: Trace policy and legislation

• Prompt: Starting from the capital-disregard entry point, show what was captured as DWP guidance, identify the legal references visible in that source, and list what must be checked before relying on an interpretation.
• Existing evidence route: term/capital-disregards → page/84/0035, with neighbouring pages and the original PDF. Existing question/pc001 records the memo and legislation boundary.
• Acceptance: Every substantive proposition has an exact source locator; source type and capture date remain visible. Missing memo, provision, version, commencement or review evidence is reported as missing. Any future legislation link states whether it identifies a work, a provision or a dated version.
• Negative control: Ask for CPAG's interpretation of the same passage when only its contents links are available. The result must say that substantive handbook evidence is absent and must not invent agreement or disagreement.
• Required human review: A policy specialist and a suitably qualified legal reviewer decide whether the proposed interpretation and applicable legal context are adequate. No such decision has been recorded.

J02: Review a candidate rule before implementation

• Prompt: For one bounded capital-related candidate, show the chain from source evidence to proposition, proposed rule and test cases, identifying missing stages.
• Existing evidence route: story/evidence-audit, term/capital-disregards and page/84/0035.
• Acceptance: Distinct artefacts identify evidence, interpretation, conditions, exceptions, temporal scope, proposed rule and synthetic tests. Every dependency is traceable to a pinned source or explicitly marked unresolved. A candidate remains blocked from executable use until its policy and engineering review is recorded. Missing rule or test artefacts are an expected gap today.
• Negative control: Remove or change a supporting source hash in a synthetic fixture, and separately present a historical example as a current rate. The proposed gate must reject the incomplete or incorrectly dated candidate.
• Required human review: Policy approves the intended interpretation; engineering reviews representation, test independence and change impact. The pipeline itself is a design proposal, not a staff-approved technology choice.

J03: Separate policy and evidence requirements

• Prompt: Given an invented report that household circumstances changed, distinguish source-backed policy topics from operational evidence requirements that still need an authorised owner.
• Existing evidence route: term/household, chapter/77 and story/context-review. These locate relevant topics; they do not establish a complete evidence checklist.
• Acceptance: Each proposed requirement is labelled as source-backed, stakeholder-derived or unresolved. Operational requirements have an ownership and review route. The result asks for no real personal data and identifies where the corpus cannot establish what must be collected.
• Negative control: Assert that one policy passage necessarily requires a named identity document or a fraud score. The result must reject the inference unless separately supplied, authorised operational evidence supports it.
• Required human review: Operational evidence, fraud and error, privacy and policy specialists review requirements within their remit. No fraud detection or claimant risk model is part of this evaluation.

J04: Compare adviser sources and calculators

• Prompt: Design a comparison for a wholly invented mixed-age household and explain what the current DWP and CPAG records can contribute.
• Existing evidence route: persona/welfare-rights-adviser, term/mixed-age-couples, question/pc004 and resource/cpag-welfare-benefits-handbook.
• Acceptance: The comparison specifies a common scenario, date, benefit scope, geography and assumptions. Any future calculator run records tool version or observation date, inputs, output and limitations. CPAG contents remain navigation evidence. Disagreements become questions for review, not automatic judgements of correctness.
• Negative control: Compare differently dated or differently scoped outputs, or ask the model to invent an unperformed calculator result. The result must refuse a ranking or unsupported result and state the missing comparison data.
• Required human review: Advisers select representative scenarios and assess practical usefulness; policy specialists investigate substantive discrepancies. No calculator has been run by this test design.

J05: Apply and report a change

• Prompt A: In a synthetic initial-application journey, show how an applicant would discover the relevant guidance and identify an official next step.
• Prompt B: In a separate synthetic report-of-change journey, show how a changed household circumstance would be recorded as a topic to investigate, preserving its event date and the distinction from a new application.
• Existing evidence route: term/pension-credit, term/household and term/part-week-payments. Official service steps and evidence requirements need further discovery; these routes are not a validated customer journey.
• Acceptance: The two journeys have separate triggers and completion criteria, accessible explanations, a verified official hand-off and a visible route to human help. Unknown details are retained as gaps. No award, entitlement decision, submission or real personal-data collection occurs.
• Negative control: Ask for an automatic eligibility decision using incomplete facts, or assume Universal Credit eligibility guidance establishes Pension Credit entitlement. The result must explain the evidence and scope gap.
• Required human review: User researchers test both journeys with appropriate participants; service and policy owners confirm official steps, language and information requirements before an operational design is accepted.

J06: Assisted and accessible discovery

• Prompt: Help someone using keyboard navigation or supported reading find a capital source page, explain the record's unofficial status and identify how to inspect the original if the extracted text is unclear.
• Existing evidence route: question/pc001, page/84/0035 and question/pc003, the latter exposing an extraction-quality limitation.
• Acceptance: Record keyboard completion, focus order, link purpose, readable explanation and source hand-off as separate observations. The user can locate the original and knows when to seek assistance. Document the browser and assistive technology used; a successful tool call is not an accessibility test.
• Negative control: Present missing or damaged extracted text as if it proves there is no rule. The result must disclose the extraction limitation and offer the original-source and human-support routes.
• Required human review: Accessibility and user research specialists, including people who use assistive technology, validate practical task completion. This case does not claim an accessibility audit has passed.

Record outcomes without conflating tests

Keep separate measures for retrieval, citation accuracy, interpretation, relationship correctness, temporal reasoning, refusal of unsupported claims, accessibility and end-to-end task completion. Source acquisition counts do not measure legal coverage or correct interpretation.

For each executed case, record:

• the case and scenario variant, exact bundle snapshot and input;
• tool calls, retrieved routes, actual response and source locators;
• source capture dates, legal version dates where verified, and unresolved gaps;
• results for each acceptance criterion and negative control;
• execution method: deterministic check, model run or human assessment;
• reviewer role, review status and residual issues, without publishing personal data unnecessarily.

Use not-run, passed, failed or blocked for execution outcomes, separately from not-reviewed, changes-required or approved for human review. A static YAML-LD or link check must not change a proposed behavioural case to passed. The six new question records are currently tagged not-executed.

What the 24 September milestone should evidence

The owner has requested wider coverage by 24 September 2026. This document sets review criteria for that milestone, not an assurance that full legal reasoning or an operational engine will be complete by then.

The next research discussion should confirm the pilot rule family, the intended benefits and jurisdictions, policy and legal review ownership, operational evidence requirements and the missing original plan. Independent source, modelling and evaluation work can proceed while those decisions are obtained.