34. Reference ledger engine, concurrency, partitioning, and retention

There is no universal global sequence
Cross-shard order is a partial order expressed through recorded causality, transaction/saga relations and a checkpoint vector. Fabricating one global counter creates a hot lock and a false temporal claim.

PostgreSQL reference DDL - compact v1.4 skeleton
CREATE TABLE stream_head (
  tenant_id uuid, stream_id uuid, logical_shard_id text,
  last_sequence bigint CHECK(last_sequence>=0), last_record_digest bytea,
  writer_epoch bigint CHECK(writer_epoch>=0),
  PRIMARY KEY(tenant_id,stream_id));

CREATE TABLE shard_head (
  ledger_id uuid, tenant_id uuid, logical_shard_id text, chain_epoch bigint,
  last_commit_sequence bigint CHECK(last_commit_sequence>=0),
  last_receipt_digest bytea NOT NULL, publisher_epoch bigint,
  shard_map_version text,
  PRIMARY KEY(ledger_id,tenant_id,logical_shard_id,chain_epoch));

CREATE TABLE record_ledger (
  ledger_id uuid, tenant_id uuid, logical_shard_id text, chain_epoch bigint,
  commit_sequence bigint CHECK(commit_sequence>0), stream_id uuid,
  stream_sequence bigint, record_id uuid, canonical_bytes bytea,
  record_digest bytea,
  PRIMARY KEY(ledger_id,tenant_id,logical_shard_id,chain_epoch,commit_sequence),
  UNIQUE(tenant_id,record_id), UNIQUE(tenant_id,stream_id,stream_sequence));

CREATE TABLE ingest_receipt (
  receipt_id uuid PRIMARY KEY, ledger_id uuid, tenant_id uuid,
  logical_shard_id text, chain_epoch bigint, commit_sequence bigint,
  previous_receipt_digest bytea, receipt_digest bytea, canonical_event_id uuid,
  UNIQUE(ledger_id,tenant_id,logical_shard_id,chain_epoch,commit_sequence));

Outbox and projection checkpoints reuse the same logical-shard/epoch/commit key.
Appendix O defines every required column, digest and atomicity constraint.

One-record append with fencing, idempotency and outbox
append(ctx, receivedBytes):
  parsed, canonicalBytes, recordDigest = validate_and_canonicalize(receivedBytes)
  requestDigest = H(domain="REQUEST", ctx.operationFamily, canonicalBytes)

  retry SERIALIZABLE transaction on SQLSTATE 40001 with bounded jitter:
    INSERT idempotency_binding(scoped key, requestDigest, status='STARTED')
      ON CONFLICT DO NOTHING
    binding = SELECT idempotency_binding FOR UPDATE by scoped key
    if binding.request_digest != requestDigest: fail EQUIVOCATION
    if binding.status != 'STARTED' or binding.receipt_id is not null:
      return original receipt or in-progress status

    head = SELECT stream_head FOR UPDATE by tenantId, streamId
    require ctx.writerEpoch == head.writer_epoch
    require ctx.expectedStreamSequence == head.last_sequence
    nextSeq = head.last_sequence + 1
    shard = stable_shard(ctx.tenantId, ctx.streamId)
    nextPosition = allocate_shard_position(shard)  # gaps allowed on rollback

    authorize exact recordDigest and current context; fail closed on error
    INSERT record_ledger, subject/relation indexes, ingest_receipt, outbox_event
    UPDATE idempotency_binding SET status='COMMITTED', receipt_id=receiptId
    UPDATE stream_head SET last_sequence=nextSeq, last_record_id=recordId,
           last_receipt_digest=receiptDigest WHERE writer_epoch=ctx.writerEpoch
    COMMIT

  publisher reads committed outbox rows, emits at least once, and marks published.
  Consumer deduplicates by eventId and advances checkpoint with its state atomically.

Hash-chain and Merkle checkpoint profile
receiptDigest[n] = SHA256(
  "RTRACER\0INGEST-RECEIPT\0v1\0" ||
  canonical(receipt_without_digest_and_signature)
)

receipt[n].previousReceiptDigest = receiptDigest[n-1] within one ledgerId + tenantId + logicalShardId + chainEpoch. ShardHead stores the only chain tail; commitSequence is gap-free and transactional.
Every 65,536 committed receipts, emit LedgerCheckpoint {
  shardId, firstCommitSequence, lastCommitSequence, merkleRoot, terminalReceiptDigest,
  recordCount, createdAt, signerKeyEpoch, backupSnapshotRefs[]
}

Daily IntegrityScrub recomputes sampled leaves, complete recent windows and
all checkpoint chains; mismatch quarantines the shard from new projection release.

Append-only does not mean retain every secret forever
The immutable property applies to the audit of decisions and relationships. Privacy-sensitive bytes can be separately encrypted, minimized, redacted and destroyed under an authorized disposal record. A tombstone must not preserve the deleted value in an index, log message, cache key or error string.