# Validation, Provenance And Catalogue Standards

Semantic meaning does not answer every publication question. A useful pack
also needs to be structurally valid, traceable to sources and understandable
to other catalogue systems.

## Four Questions

These questions are related but independent:

1. **Syntax:** Can the file be parsed?
2. **Structure:** Does it match the expected data shape?
3. **Semantics:** What do its identified terms mean?
4. **Evidence:** Where did each important claim come from?

A JSON file can be valid syntax and still omit its title. A record can pass a
schema and still contain an incorrect publisher. A correct claim can still be
unusable if no one can trace it to a source.

## JSON Schema

JSON Schema describes the permitted structure of JSON data.

A simplified rule might say:

```json
{
  "type": "object",
  "required": ["id", "title"],
  "properties": {
    "id": { "type": "string", "minLength": 1 },
    "title": { "type": "string", "minLength": 1 },
    "tags": {
      "type": "array",
      "items": { "type": "string" }
    }
  }
}
```

It can check:

- object, array, string, number and boolean types;
- required properties;
- allowed values and patterns;
- numeric or length bounds;
- nested combinations;
- references to reusable definitions.

The bundle-wiki profile includes schemas for descriptors, presentations and
provider datapacks. JSON Schema operates on the JSON representation.

Passing a schema means the value fits the declared structural rules. It does
not prove that a source is authoritative or a URL is safe to contact.

## SHACL

Shapes Constraint Language, or **SHACL**, validates RDF graphs. A SHACL shape
can say that nodes targeted as datasets must have:

- exactly one title;
- a title with a suitable datatype;
- a publisher that is an identified node;
- evidence for an inferred statement;
- a value drawn from a controlled scheme.

A validation result can identify:

- the focus node;
- the shape and constraint;
- the property path;
- severity;
- a human message;
- the source artefact.

Publishing results as data makes remediation reviewable and lets the Explorer
eventually present a validation view.

## Closed Publication Rules And Open-World Meaning

SHACL can check this particular data graph against a declared contract.
JSON Schema can check a JSON document. These are effectively closed
publication questions: did this artefact include what its profile requires?

RDFS and OWL reasoning use open-world semantics: missing information can be
unknown.

Therefore:

- a missing required title can fail the publication profile;
- the failure does not prove that the described thing has no title in the
  world;
- an RDFS range declaration can infer a type;
- the range declaration is not a replacement for a validation shape.

The Explorer architecture reports validation and inference separately.

## Conformance And Validation

**Validation** is running checks.

**Conformance** is meeting a named specification or profile, including its
required rules and versions.

A project can borrow fields from DCAT without claiming DCAT conformance. The
repository currently describes parts of its alignment as
“standards-alignable, not standards-conformant.” This is more honest than
making a broad claim from a few matching field names.

Conformance statements should name:

- the exact profile and version;
- required and optional features;
- known deviations;
- the validator and date or snapshot tested.

## Provenance

**Provenance** is information about origin and transformation:

- which source supplied a value;
- when it was observed;
- which activity normalised or inferred it;
- which software or person performed the activity;
- which prior entities it used;
- which output it generated.

Provenance supports trust decisions; it does not dictate them. The same
traceable source may be authoritative for one question and unsuitable for
another.

## PROV

The W3C PROV data model organises provenance around:

- **Entity** — a source document, dataset, assertion or generated artefact;
- **Activity** — harvesting, parsing, normalisation, mapping, inference or
  validation;
- **Agent** — a person, organisation or software system responsible for an
  activity.

Example:

```text
catalogueRow       --used by----------> normalizationActivity
normalizationTool  --associated with--> normalizationActivity
normalizedRecord   --generated by-----> normalizationActivity
```

PROV-O is the RDF vocabulary for expressing this model. Qualified relations
can describe roles and influences when a simple edge is not enough.

## Assertion-Level Evidence

Whole-record provenance is sometimes too coarse. A record can combine:

- an official title;
- a normalised licence;
- a model-derived topic;
- an inferred publisher type.

Each relationship or important field can carry:

- assertion status;
- evidence type;
- one or more evidence identifiers;
- confidence method and score where appropriate;
- observation time;
- derivation activity.

That prevents one source link from appearing to justify every generated value.

## DCAT

Data Catalog Vocabulary, or **DCAT**, supplies RDF terms for:

- catalogues;
- catalogue records;
- datasets;
- data services;
- distributions;
- publishers, themes, licences and access URLs.

A dataset is the conceptual data collection. A distribution is a particular
available representation, such as a CSV download or API endpoint. Treating
every file URL as a separate dataset loses that distinction.

DCAT 3 is the semantic baseline used when OKF records describe catalogue
material.

## DCAT-AP

DCAT Application Profile, or **DCAT-AP**, narrows and extends DCAT for European
data portals. An application profile states which terms, cardinalities and
controlled vocabularies a community expects.

The repository's crosswalk maps OKF fields to DCAT-AP so export gaps are
visible. A mapping is not the same as complete conformance.

## DQV

Data Quality Vocabulary, or **DQV**, describes quality measurements,
annotations, policies and metrics alongside datasets.

A metadata completeness score needs:

- the metric definition;
- calculation method;
- applicable records;
- observation time;
- missing-value treatment.

Without those, “quality: 82” is a decorative number rather than interoperable
evidence.

## OpenAPI

OpenAPI describes HTTP APIs:

- paths and operations;
- request parameters and bodies;
- response schemas;
- authentication schemes;
- server addresses;
- human documentation.

DCAT describes a data service in a catalogue; OpenAPI describes how to call
that service. One API record may link to both catalogue metadata and an
OpenAPI contract.

The UK Government APIs pack records standards gaps rather than fabricating an
OpenAPI operation when the source does not provide enough detail.

## Other Standards In The Corpus

The original research corpus covers standards that underpin agent-ready
infrastructure:

- OAuth, PKCE, DPoP and mutual TLS for delegated or sender-constrained access;
- OpenAPI, GraphQL, gRPC and AsyncAPI for interfaces;
- MCP and A2A for tool and agent interaction;
- Arazzo for multi-operation workflows;
- OPA for policy decisions;
- OpenTelemetry for traces, metrics and logs;
- HTTP Message Signatures and workload identity for authenticity;
- JSON Schema and structured outputs for machine-readable contracts.

These standards do not all run inside the Explorer. They are subject matter in
the sample corpus and useful reference points for its evidence model.

## Validation In The Publication Pipeline

Different checks catch different errors:

| Check | Example failure |
|---|---|
| Markdown/link parsing | A link targets no known record |
| JSON Schema | A descriptor lacks `entrypoints.data_manifest` |
| Semantic expansion | A JSON-LD context or identifier is invalid |
| SHACL | A governed concept lacks its required scheme |
| Cross-file invariant | A manifest count differs from its chunks |
| Integrity | Artefact bytes do not match the declared hash |
| Browser contract test | A valid pack cannot be selected or navigated |
| Evaluation question | Search misses evidence needed for a real task |

No single validator replaces the others.

## Evidence Before Fluency

The Explorer's final purpose is not to produce the smoothest description. It
is to let a reader or agent find an answer whose sources, transformations and
limits remain inspectable.

## Next

[Explorer views and presentation](09-explorer-views-and-presentation.md)
explains how one evidence model becomes several usable interfaces without
changing its meaning.