Validation, Provenance And Catalogue Standards#

Semantic meaning does not answer every publication question. A useful pack also needs to be structurally valid, traceable to sources and understandable to other catalogue systems.

Four Questions#

These questions are related but independent:

1. Syntax: Can the file be parsed?
2. Structure: Does it match the expected data shape?
3. Semantics: What do its identified terms mean?
4. Evidence: Where did each important claim come from?

A JSON file can be valid syntax and still omit its title. A record can pass a schema and still contain an incorrect publisher. A correct claim can still be unusable if no one can trace it to a source.

JSON Schema#

JSON Schema describes the permitted structure of JSON data.

A simplified rule might say:

{
  "type": "object",
  "required": ["id", "title"],
  "properties": {
    "id": { "type": "string", "minLength": 1 },
    "title": { "type": "string", "minLength": 1 },
    "tags": {
      "type": "array",
      "items": { "type": "string" }
    }
  }
}

It can check:

• object, array, string, number and boolean types;
• required properties;
• allowed values and patterns;
• numeric or length bounds;
• nested combinations;
• references to reusable definitions.

The bundle-wiki profile includes schemas for descriptors, presentations and provider datapacks. JSON Schema operates on the JSON representation.

Passing a schema means the value fits the declared structural rules. It does not prove that a source is authoritative or a URL is safe to contact.

SHACL#

Shapes Constraint Language, or SHACL, validates RDF graphs. A SHACL shape can say that nodes targeted as datasets must have:

• exactly one title;
• a title with a suitable datatype;
• a publisher that is an identified node;
• evidence for an inferred statement;
• a value drawn from a controlled scheme.

A validation result can identify:

• the focus node;
• the shape and constraint;
• the property path;
• severity;
• a human message;
• the source artefact.

Publishing results as data makes remediation reviewable and lets the Explorer eventually present a validation view.

Closed Publication Rules And Open-World Meaning#

SHACL can check this particular data graph against a declared contract. JSON Schema can check a JSON document. These are effectively closed publication questions: did this artefact include what its profile requires?

RDFS and OWL reasoning use open-world semantics: missing information can be unknown.

Therefore:

• a missing required title can fail the publication profile;
• the failure does not prove that the described thing has no title in the world;
• an RDFS range declaration can infer a type;
• the range declaration is not a replacement for a validation shape.

The Explorer architecture reports validation and inference separately.

Conformance And Validation#

Validation is running checks.

Conformance is meeting a named specification or profile, including its required rules and versions.

A project can borrow fields from DCAT without claiming DCAT conformance. The repository currently describes parts of its alignment as “standards-alignable, not standards-conformant.” This is more honest than making a broad claim from a few matching field names.

Conformance statements should name:

• the exact profile and version;
• required and optional features;
• known deviations;
• the validator and date or snapshot tested.

Provenance#

Provenance is information about origin and transformation:

• which source supplied a value;
• when it was observed;
• which activity normalised or inferred it;
• which software or person performed the activity;
• which prior entities it used;
• which output it generated.

Provenance supports trust decisions; it does not dictate them. The same traceable source may be authoritative for one question and unsuitable for another.

PROV#

The W3C PROV data model organises provenance around:

• Entity — a source document, dataset, assertion or generated artefact;
• Activity — harvesting, parsing, normalisation, mapping, inference or validation;
• Agent — a person, organisation or software system responsible for an activity.

Example:

catalogueRow       --used by----------> normalizationActivity
normalizationTool  --associated with--> normalizationActivity
normalizedRecord   --generated by-----> normalizationActivity

PROV-O is the RDF vocabulary for expressing this model. Qualified relations can describe roles and influences when a simple edge is not enough.

Assertion-Level Evidence#

Whole-record provenance is sometimes too coarse. A record can combine:

• an official title;
• a normalised licence;
• a model-derived topic;
• an inferred publisher type.

Each relationship or important field can carry:

• assertion status;
• evidence type;
• one or more evidence identifiers;
• confidence method and score where appropriate;
• observation time;
• derivation activity.

That prevents one source link from appearing to justify every generated value.

DCAT#

Data Catalog Vocabulary, or DCAT, supplies RDF terms for:

• catalogues;
• catalogue records;
• datasets;
• data services;
• distributions;
• publishers, themes, licences and access URLs.

A dataset is the conceptual data collection. A distribution is a particular available representation, such as a CSV download or API endpoint. Treating every file URL as a separate dataset loses that distinction.

DCAT 3 is the semantic baseline used when OKF records describe catalogue material.

DCAT-AP#

DCAT Application Profile, or DCAT-AP, narrows and extends DCAT for European data portals. An application profile states which terms, cardinalities and controlled vocabularies a community expects.

The repository's crosswalk maps OKF fields to DCAT-AP so export gaps are visible. A mapping is not the same as complete conformance.

DQV#

Data Quality Vocabulary, or DQV, describes quality measurements, annotations, policies and metrics alongside datasets.

A metadata completeness score needs:

• the metric definition;
• calculation method;
• applicable records;
• observation time;
• missing-value treatment.

Without those, “quality: 82” is a decorative number rather than interoperable evidence.

OpenAPI#

OpenAPI describes HTTP APIs:

• paths and operations;
• request parameters and bodies;
• response schemas;
• authentication schemes;
• server addresses;
• human documentation.

DCAT describes a data service in a catalogue; OpenAPI describes how to call that service. One API record may link to both catalogue metadata and an OpenAPI contract.

The UK Government APIs pack records standards gaps rather than fabricating an OpenAPI operation when the source does not provide enough detail.

Other Standards In The Corpus#

The original research corpus covers standards that underpin agent-ready infrastructure:

• OAuth, PKCE, DPoP and mutual TLS for delegated or sender-constrained access;
• OpenAPI, GraphQL, gRPC and AsyncAPI for interfaces;
• MCP and A2A for tool and agent interaction;
• Arazzo for multi-operation workflows;
• OPA for policy decisions;
• OpenTelemetry for traces, metrics and logs;
• HTTP Message Signatures and workload identity for authenticity;
• JSON Schema and structured outputs for machine-readable contracts.

These standards do not all run inside the Explorer. They are subject matter in the sample corpus and useful reference points for its evidence model.

Validation In The Publication Pipeline#

Different checks catch different errors:

No single validator replaces the others.

Evidence Before Fluency#

The Explorer's final purpose is not to produce the smoothest description. It is to let a reader or agent find an answer whose sources, transformations and limits remain inspectable.

Next#

Explorer views and presentation explains how one evidence model becomes several usable interfaces without changing its meaning.