On this page· 4
Specification
- Revision 6.0
- file format 6
- Whole specification on one page
The rank is the canonical sort position. The primary id is the record's identity within its kind; pad20(n) means n in decimal, left-padded with 0 to 20 digits. The field list is the v1 schema; ? marks an OPTIONAL member, all others are REQUIRED. Every kind also carries v? and extra.
| Rank | k |
Primary id | v1 members |
|---|---|---|---|
| 0 | meta |
created_at "|" source |
created_at, source, tool?, description?, ngdb_generation? (u64) |
| 1 | payload |
empty string | embedded (bool), csdt_file_checksum? (u32, always emitted, default 0), sections? (array of {index u32, section_type u16, record_count u64, dtype? string}, always emitted, default []), refs? (array of string, always emitted, default []) |
| 2 | csdt_ref |
ref_id |
ref_id, path, shard_set_id? (u32), file_checksum (u32), source_hash, section_type (u16), section_index (u32), record_range? ([u64, u64]) |
| 3 | ent |
id |
id, t, name?, salience? (f32), stub? (bool, always emitted, default false), properties?, prov? |
| 4 | rel |
src "|" tgt "|" kind |
src, tgt, kind, properties?, prov? |
| 5 | hyper |
id |
id, members (array of string), kind, properties? |
| 6 | embref |
entity_id "|" target-key |
entity_id, target (§5.4), annotations? |
| 7 | prov |
subject "|" method "|" pad20(unix_secs) |
subject, agent, method, unix_secs (u64), inputs?, notes? |
| 8 | grounded_answer |
answer_id |
Application kind. |
| 9 | citation |
source_answer_id "|" entity_id "|" pad20(span.start) |
Application kind. |
| 10 | clause |
clause_id |
Application kind. |
| 11 | knob_set |
scope "|" key "|" pad20(unix_secs) |
Application kind. |
| 12 | discovery |
gid |
Application kind. |
| 13 | promo |
content_hash "|" pad20(order) |
Application kind. |
| 14 | codebook_manifest |
codebook_id |
Application kind. |
| 15 | session |
session_id |
Application kind. |
| 16 | sev |
pad20(step) "|" t |
Application kind. |
| 17 | session_end |
pad20(end_unix) |
Application kind. |
| 18 | outcome |
identity form (§10.3) of outcome |
Application kind. |
Members without a stated type are JSON strings; u32, u64 are non-negative JSON integers in range.
Changing a typed kind's schema incompatibly requires raising its v (§12.2). Adding an OPTIONAL member at the same v is compatible, because older readers keep it in extra.
These kind strings are reserved, but have no typed parser in this revision. Readers MUST treat them as opaque (§4.4).
mut
Ten further kind names are reserved and are specified separately. A reader keeps any record whose kind it does not know as an opaque record (§4.4).
A reserved kind is promoted to the typed registry only by a spec revision that adds it to §5.1 with the same v1 schema.
Kind names that begin with x- belong to applications: no revision of this specification adds one to the typed kinds (§5.1) or the reserved kinds (§5.2).
mut is one entry of an ordered mutation log. The schema below is the wire form of mut v1.
| Member | Type | Rule |
|---|---|---|
k |
string | "mut" |
v |
integer | 1 |
id |
string | the LSN in decimal, left-padded with 0 to 20 digits. MUST equal pad20(lsn) |
gen |
u64 | publication generation |
lsn |
u64 | log sequence number within gen |
op |
string | one of put_record, delete_record, or one of the op names reserved for future use |
rk |
string, OPTIONAL | kind of the affected record. REQUIRED for put_record and delete_record |
rid |
string, OPTIONAL | primary id (§5.1) of the affected record. REQUIRED for put_record and delete_record |
rec |
object, OPTIONAL | put_record only: the complete affected record as its own JSONL object, including its k. Its kind and primary id MUST equal rk and rid. delete_record MUST NOT carry rec |
| other | any | members of the reserved ops, preserved. None of them may be named id |
Emitted member order (under §10.1): k, v, id, gen, lsn, op, rk, rid, rec, then other members.
Rules the member table cannot express:
- One file holds one
gen. LSNs restart whengenchanges, so two generations in one file would shareids and merge would drop records. - LSNs within a
genare gap-free and are applied in LSN order. Becausemutis opaque, it sorts after every typed kind, byid, which is LSN order. - A mutation log is ordered, not a set. Producers MUST NOT combine logs with merge (§8).
rk/rididentify records, neverid: an opaque record's identity readsidfirst (§5.3), so an entity id there would collapse a put and a delete of the same record.
For every record:
- kind = the
kstring. - rank = §5.1 rank when
kis a typed kind string (this includes an opaque record whosekis a typed kind at a higherv), otherwise the maximum rank, placed after all typed kinds. - primary id = §5.1 for typed records. For an opaque record: the value of the first of
id,ref_id,session_id,answer_id,clause_idthat is present as a JSON string; if none is, the identity form (§10.3) of the whole object. - sort key =
(rank, kind, primary id), compared rank numerically, then kind and primary id by UTF-8 byte order. - dedup key =
(kind, primary id).
A writer MUST emit records in ascending sort key order. Records with equal sort keys keep their input order (the sort is stable). A writer SHOULD NOT emit two records with the same dedup key; readers MUST accept such files, and diff reports them (§8.2).
Note: an opaque record whose k is a typed kind at a higher v has the same dedup key as a v1 record with the same primary id. Merge treats the two as the same record (§8.1).
embref.target, session.trajectory, codebook_manifest.target and promo.curvature_refs[] are one of two JSON shapes:
{"local":{"section_index":S,"record_index":R}}: recordRof sectionSof this file's own appendix. target-key ="local|" pad10(S) "|" pad20(R).{"ext":{"ref_id":F,"record_index":R}}: recordRof the section named by thecsdt_refrecord whoseref_idisF.Rcounts from the section start, not from anyrecord_range. target-key ="ext|" F "|" pad20(R).
(pad10 pads to 10 digits.)
© 2020-2026 Cintile Inc. All Rights Reserved.
Anyone may implement this format. Copying or republishing the text of this specification requires permission from Cintile Inc.