8. Operations

Merge takes two packs a and b and a strategy, and produces one pack.

Preconditions:

1. Each input MUST pass verify steps 1 to 8 (§9) before merging.
2. Each input appendix MUST have container version byte 5 (§7.2), else appendix_legacy_version.
3. An input appendix holding a tensor catalog section MUST be rejected with merge_conflict: tensor catalogs encode section indexes internally and cannot be re-indexed.
4. An input section whose section_type is not in the CSDT registry MUST be rejected with merge_conflict.

Appendix layer:

1. Lift every section of a and b as the tuple (section_type, format_version, payload_class, flags, record_count, record_stride, dtype bytes, align_log2, data bytes).
2. The output pool is all lifted tuples, sorted ascending by that tuple in that field order (integers numerically, byte strings lexicographically), with exact duplicates removed. sections_deduped counts the removed duplicates.
3. Each input section's new index is its position in the pool. Every local binding target (§5.4) in that input's records is rewritten to the new section index; record_index is unchanged, because sections move whole.
4. The pool is written as a new version-5 container. An empty pool means no appendix.

Record layer, applied to a's records and then b's, one record at a time, keyed by dedup key (§5.3):

1. payload records from the inputs are dropped; one new payload record describing the output appendix is added at the end. Its members are embedded (true when the output has an appendix), csdt_file_checksum (the output container's file_checksum, else 0), sections (one {index, section_type, record_count} per pool entry, without dtype) and refs (the ref_id of every csdt_ref record in the output). All four are emitted even when empty, so an empty pool with no csdt_ref gives {"k":"payload","v":1,"embedded":false,"csdt_file_checksum":0,"sections":[],"refs":[]}.
2. New key: insert (records_inserted).
3. Existing record equal to the incoming one (value equality after parsing): keep, count records_skipped.
4. Both ent:
   • existing is a stub and incoming is not: take incoming, count stubs_resolved and records_updated;
   • incoming is a stub and existing is not: keep existing (records_skipped);
   • otherwise a conflict (conflicts), resolved by the strategy: higher_salience takes incoming only if its salience is strictly greater (missing counts as 0.0); latest takes incoming; union keeps existing; manual keeps existing and records the conflict as deferred. A conflict counts conflicts; when the strategy takes the incoming record it also counts records_updated, and when it keeps the existing one, records_skipped.
5. Any other kind: latest takes incoming (records_updated); every other strategy keeps existing (records_skipped).
6. The output records are written in sort key order (§5.3) with the rules of §2 to §6.

Properties:

• For inputs whose shared dedup keys carry equal records, merge(a, b) and merge(b, a) produce byte-identical files. With conflicts, the result depends on argument order under latest and higher_salience ties.
• Merge treats mut records like any opaque record (dedup by the padded LSN in id). Combining two mutation logs with merge is a producer error (§5.2.1), not something merge detects.
• The merge report keys are records_inserted, records_updated, records_skipped, stubs_resolved, conflicts, sections_merged, sections_deduped.

Diff compares two packs a (before) and b (after).

1. Index each side by dedup key (§5.3). A key that occurs more than once on either side is listed in duplicate_keys and is not otherwise compared.
2. added: keys only in b. removed: keys only in a.
3. changed: keys in both whose records differ in identity form (§10.3). Each entry carries the key and both record lines.
4. Appendix: sections are matched by section index; a section is added, removed, or changed when its section_type or its descriptor CRC32C differs.
5. Every output list is sorted by sort key (records) or by section index (sections).

Diff never reports a difference that exists only in member order, whitespace or number spelling, because it compares identity forms.

Shard produces a new pack holding a connected subset of one pack's graph. The Sharder conformance level is provisional in revision 6.0; this section is its contract.

Graph view of a pack: nodes are ent records by id; directed edges are rel records src → tgt labelled kind; an entity has an embedding when an embref record names it.

Shard specs:

• entity_centric{seed, max_depth, max_fanout?, relationship_filter?, direction, phase_coherent}: breadth-first from seed to max_depth hops, following edges in direction (outgoing, incoming, both), at most max_fanout neighbours per node when set, and only edges whose kind is in relationship_filter when set.
• search_result{entity_ids, context_depth, max_fanout?}: the listed entities plus their neighbourhood to context_depth.
• predicate{...}: entities whose fields satisfy the predicate.
• cluster{cluster_id, include_bridges}: MUST fail with shard_unsupported_spec until a cluster field exists in the record layer. It MUST NOT return an empty result instead.
• union[specs]: the union of the member results.

Output pack:

• the selected ent records, plus a stub ent (stub: true) for each edge endpoint outside the selection when the config asks for stubs;
• the rel records whose endpoints are both in the output;
• a hyper record only when every member is selected;
• embref records of selected entities, with the appendix rewritten to hold only the referenced rows and the bindings remapped;
• every other appendix section listed in dropped_sections, never dropped silently;
• a meta record whose created_at is supplied by the caller. Two runs with the same input, spec and created_at MUST produce byte-identical packs.

A shard whose quality score is below a configured minimum fails with shard_quality_below_threshold.