Large Corpora And Progressive Loading#

A person opening a collection of hundreds of thousands of records should not have to download every full record before the first useful screen appears. The large-corpus design keeps static publication while making loading progressive.

The Restaurant-Menu Analogy#

A menu does not bring every dish to the table. It gives enough information to choose, then the kitchen prepares what was selected.

A large OKF descriptor plays the role of the menu. It states what the collection is, which artefacts exist and where to find them. Search indexes and compact result records help the user choose. Full records and relationship neighbourhoods load only when needed.

Unlike a restaurant, the artefacts are built in advance; the static host is not running custom queries.

Descriptor#

The public entry point is commonly named okf-explorer.json. A simplified descriptor looks like this:

{
  "schema": "okf-explorer-large-corpus.v1",
  "kind": "okf-large-corpus",
  "title": "Example Data Catalogue",
  "snapshot": "2026-07-01",
  "entrypoints": {
    "data_manifest": "data/manifest.json",
    "analysis_overview": "data/analysis.json",
    "search_manifest": "data/search/manifest.json"
  },
  "counts": {
    "records": 120000
  }
}

The descriptor is small enough to inspect and fetch quickly. It describes the publication but does not contain every record.

Manifest#

A manifest inventories artefacts in a publication. The data manifest can point to:

• overview and analysis files;
• search indexes;
• facets;
• full-record chunks;
• resource and publisher chunks;
• relationship indexes and adjacency buckets;
• provider datapacks;
• integrity metadata.

The descriptor is the stable front door; manifests describe the files behind it.

Control Plane And Data Plane#

These terms are borrowed from distributed systems.

The control plane tells the application what exists and how to access it: descriptor, manifest, version, snapshot, counts, schemas and entrypoints.

The data plane carries the substantial content: records, search postings, relationships, resources and compressed packs.

Keeping them separate lets the Explorer make a plan from a small trusted description before transferring larger data.

This is unrelated to the AI-infrastructure corpus's “discovery-to-governance control plane,” which describes governed agent execution. The same phrase is used at two different architectural levels.

Chunks And Shards#

Both split a large artefact, but the reason for the split differs.

A chunk is a bounded group of records, often chosen by numeric position:

data/records-000.json
data/records-001.json

A shard is a partition selected by a deterministic key. Search terms might be divided by their first characters, while relationships can be divided by a hash of the selected route.

The Explorer must use the same partitioning algorithm as the builder. Published test vectors protect details such as non-ASCII identifiers from producing different buckets in Python and JavaScript.

Indexes#

An index is extra data built to answer a particular question quickly.

Examples include:

• a search lexicon mapping terms to postings;
• a facet index mapping a value to matching record numbers;
• a document map translating record IDs to numeric positions;
• a relationship adjacency index locating the edges near one route;
• sort values aligned to record positions.

Indexes duplicate selected information intentionally. The canonical record is still distinct from its search or display projection.

A facet value such as Scotland may therefore have an exact posting list without being a materialised relationship node. Explorer can show a bounded membership preview and graph from that posting, clearly labelled as index-derived. Selecting one real record then loads that record's adjacency shard. Asking a facet card to load every corpus relationship is both unnecessary and semantically wrong.

Progressive Loading#

The normal large-corpus sequence is:

1. fetch the descriptor;
2. fetch the data and search manifests;
3. render overview counts and notices;
4. load only the search shards required for a query;
5. load compact result-document chunks;
6. load a full record after selection;
7. load its resources or relationship neighbourhood on demand;
8. fetch external source material only after a deliberate user action.

This is also called lazy loading: defer work until it can provide value.

Progressive loading is not an excuse for hidden incompleteness. The interface must say when a total is exact, a lower bound or unknown, and when a result was truncated by a budget.

Compression#

Repeated text in JSON compresses well. Large artefacts may be published with gzip compression to reduce transfer size.

Compression introduces two sizes:

• compressed bytes transferred over the network;
• uncompressed bytes parsed in memory.

A small compressed file can expand greatly, so the Explorer limits the uncompressed result as well as the response. Browsers may transparently decompress HTTP content; files can also explicitly carry a .gz form.

Integrity And Snapshots#

A cryptographic hash such as SHA-256 is a short fingerprint calculated from bytes. If the bytes change, the expected hash no longer matches.

Integrity metadata helps answer:

• Did the fetched artefact match the manifest?
• Did artefacts from two different snapshots get mixed?
• Can a frozen release be verified later?

It does not prove the content was correct at the source. It proves that the bytes match the stated publication.

A snapshot is a coherent version of a pack. Mutable live URLs are convenient, while immutable release assets are better for reproducibility. The publication model can provide both.

Byte-Range Release Packs#

Static hosts and release systems can impose limits on file count or request patterns. Several immutable artefacts can be concatenated into a larger pack. An index records each artefact's offset and length.

An HTTP Range request asks for only those bytes. The Explorer verifies the selected slice before parsing it. This keeps a static, immutable release usable without downloading the entire pack for one record.

Range packs are an optimisation and mirror. The logical artefact paths remain the data contract.

Registries#

A registry is a discoverable list of bundle descriptors. It can provide examples and labels, but it is not a central database through which every pack must pass.

The Explorer can also remember recently entered bundle URLs in browser storage. Registry entries and personal history help discovery; the bundle URL remains the actual source selection.

Provider Datapacks#

A provider datapack attaches a governed snapshot or a bounded, reviewed reference to records selected by declared rules.

It distinguishes:

• content included in the pack's snapshot;
• information reviewed from an external provider;
• known drift between those states;
• an action that deliberately opens or contacts the provider.

The Explorer must not silently present live external data as though it were part of the governed snapshot.

Failure Is Part Of The Contract#

The interface needs useful states for:

• missing or invalid descriptors;
• unsupported schema versions;
• a missing shard;
• integrity mismatch;
• timeouts or CORS failure;
• decompression or parsing limits;
• stale snapshots;
• a source that has moved to a new descriptor.

Recovery should preserve enough context for the user to retry, inspect the source or move to a documented target.

Small Versus Large#

Neither is “more OKF.” They are publication shapes for different scales.

Next#

Search, filters, facets and context explains the most important indexes and how their results should be interpreted.