30. Runtime architecture, persistence, replay, scale, and recovery

The runtime uses one provenance spine and several explicitly subordinate stores. This is not accidental polyglot persistence: semantic records, high-volume telemetry, immutable artifacts, full-text discovery, graph traversal and low-latency feeds have different access patterns. Only the canonical ledger and artifact digests are required to rebuild the others.

Specialized stores accelerate access; none may silently mutate semantic history.

#30.1 Store responsibility matrix

#30.2 Aggregate stream head and fenced writer

• Only one writer epoch is accepted for a stream at a time; lease expiry alone is insufficient without fencing.
• RTracer promises no global semantic order across independent streams. Cross-stream views expose their contributing checkpoints.
• A hash chain is tamper-evident, not magically tamper-proof. Independent receipt anchoring, backups and comparison are still required.
• Cross-stream workflows are explicit sagas with durable intents, compensations and reconciliation—not hidden distributed transactions.

#30.3 Atomic append and transactional outbox

Atomic append and canonical outbox - v1.4 pseudocode
BEGIN
  lock idempotency_binding(tenant, principal, operation, key)
  if existing.request_digest == request_digest: return existing result
  if existing and digest differs: quarantine EQUIVOCATION
  lock ShardHead rows in canonical shard-key order
  lock stream_head rows in canonical stream-key order
  recheck writer, policy, registry and shard-map epochs
  require expected_stream_seq == stream_head.last_stream_seq
  nextStreamSequence = stream_head.last_stream_seq + 1
  nextCommitSequence = shard_head.last_commit_sequence + 1
  build receipt using shard_head.last_receipt_digest
  insert record, receipt, exactly one canonical admission event and outbox row
  advance stream_head and ShardHead in the same transaction
COMMIT

Publisher emits the canonical lane by tenant + logicalShardId + chainEpoch in
commitSequence order. Consumer deduplication, projection mutation and checkpoint
advance commit atomically over a contiguous applied prefix.
BEGIN
  lock idempotency_binding(tenant, principal, key)
  if existing.request_digest == request_digest: return existing.result
  if existing and digest differs: quarantine EQUIVOCATION
  lock stream_head(tenant, stream_id)
  require writer_epoch == stream_head.writer_epoch
  require expected_stream_seq == stream_head.last_stream_seq
  validate canonical record and authorization at policy checkpoint
  assign tenant ledger_seq and next stream_seq
  insert record_ledger, record_subject, record_relation
  insert idempotency_binding(request_digest, result_record_id)
  insert outbox_event(ledger_seq, canonical_event_digest)
  advance stream_head(stream_seq, record_id, record_digest, receipt_digest)
COMMIT

Publisher claims outbox rows, emits at-least-once, records acknowledgement.
Consumers deduplicate by tenant + ledger_seq + event kind and checkpoint only
after the projection mutation is durable.

Exactly-once is an outcome, not a transport promise
The platform assumes messages can be duplicated, delayed and reordered. Idempotent writes, immutable sequence IDs, deterministic consumers and durable checkpoints make the resulting projection effectively once-applied.

#30.4 Projection contract

#30.5 Partitioning and tenancy

#30.6 Consistency model

#30.7 Capacity and performance model

Capacity planning starts with record and artifact classes, not vehicle count alone. One museum car may produce a few records per month; a race car may produce millions of samples per session; a public vehicle agent may produce many reads with few writes.

#30.8 Service objectives and error budgets

#30.9 Backup, disaster recovery, and integrity scrubbing

1. Continuously replicate canonical ledger logs and object manifests to an administratively independent failure domain.
2. Back up schema/vocabulary/pack registries and encryption/key metadata under separate access controls; encrypted data without recoverable authorized keys is lost.
3. Verify backups by scheduled restore into an isolated environment, then replay projections and compare known digests/checkpoints.
4. Run periodic artifact integrity scrubs against stored digests; repair from replica or mark unavailable with an incident record.
5. Maintain a documented failover decision, split-brain prevention, recovery sequence and reconciliation path for writes accepted near outage boundaries.
6. Record recovery actions and data-loss boundaries as operational receipts; do not imply perfect continuity when RPO was exceeded.
7. Exercise tenant-level export and restoration so a local failure or commercial migration is not dependent on the original platform instance.