Course contents
On this page
A complete architecture walkthrough — extended reference
Original detailed coverage and citations accompanying the focused a complete architecture walkthrough lesson.
This reference preserves the earlier detailed chapter and its citations. Start with the guided explanation, then use these sections for additional mechanisms and qualifications. References to the original lesson sequence are historical.
Part 8: Architectural case studies and synthesis
Prerequisites: Lesson 47 and the preceding foundations.
Learning outcomes
- Carry explicit invariants through storage, logging, concurrency, replication, partitioning, and operational evolution.
- Explain how the contended marketplace, multitenant service, and realtime world force different architectural branches.
- Produce a coherent capacity and failure argument that accounts for hot resources, durable acknowledgments, recovery, and overload.
Begin with one invariant and an explicit failure model
The reference engine in this lesson is a teaching architecture, not a production-ready implementation. It evolves through concrete stages so each new mechanism answers a named problem. The invariant is simple: a unique item has exactly one owner or one valid world location. A transaction can transfer an item, consume ingredients to create another item, or reserve an item for purchase.
Three applications reuse that invariant differently. A contended marketplace concentrates buyers on popular listings. A multitenant service mostly updates independent tenant-owned records. A shared world updates authoritative state and delivers selected changes continuously. Their access patterns diverge even when their tables look similar.
The initial failure model covers one process crash with persistent local storage. Later stages add machine loss, network partitions, leadership changes, partial distributed transactions, and regional failures. Adding a mechanism without naming the new failure or workload would make the architecture harder to justify. Each stage preserves earlier correctness obligations while introducing new ones.
records + indexes -> atomic log/recovery -> retry identity -> concurrency
-> durable replication -> partition ownership -> bounded delivery/operations
|-> marketplace: exact contended decisions
|-> SaaS: tenant-local independence
|-> world: local interaction and selective fan-outFigure: Responsibilities accumulate while workload-specific boundaries diverge. Every branch retains its original correctness obligations.
Stage 1: represent authoritative facts once
The first engine stores records keyed by stable identifiers. Ownership lives in one authoritative record rather than duplicated fields maintained by separate services. Secondary indexes map owner identifiers to item identifiers for efficient inventory reads.
Item(item_id, owner_kind, owner_id, version)
Operation(actor_id, request_id, outcome)
OwnerIndex(owner_kind, owner_id, item_id)This is a logical sketch, not a prescribed wire format. The index must agree with the item record. A transfer removes the old index entry, adds the new entry, increments the record version, and records the request outcome. The invariant now includes consistency between primary and secondary representations.
Initially, one execution thread owns all mutation. That makes physical memory access simple, but a crash can erase all recent work. The engine has not solved durability merely because it can update several structures without interference. It also needs bounded inputs: maximum key size, value size, and operation complexity prevent malformed requests from exhausting memory.
At this stage, the best performance work is straightforward: use indexed access rather than full scans, avoid unnecessary copying, and keep stable identities. There is no reason to introduce distributed coordination before the application requires it.
Stage 2: introduce atomic recovery
A multi-record in-memory mutation can be interrupted by a crash. The engine therefore needs a recoverable representation of transaction boundaries. One possible teaching design stages private writes, appends a complete transaction record with identity and checksum, durably flushes according to policy, installs the committed writes, and acknowledges success. Actual engines can choose different installation and logging strategies while satisfying their recovery rules.
validate request and construct private write set
encode transaction with bounded lengths and checksum
append transaction record to log
make required log prefix durable
publish writes as one committed state transition
record / return outcome for stable request identityFigure: A simplified log-first teaching path. Publication and acknowledgment ordering depend on the full recovery protocol; the drawing does not substitute for one.
Publication uses a generation or equivalent reader barrier so item and secondary-index changes become visible together; readers cannot observe a half-installed transaction. Recovery reads complete valid records, identifies the durable committed prefix under the design, and reapplies transactions idempotently. A truncated final record is not treated as a successful transaction. Corruption in the middle of an acknowledged history cannot simply be ignored as though it were an incomplete tail.
Write-ahead logging is a general technique with concrete ordering rules. PostgreSQL’s documentation illustrates why data-page persistence must respect the log dependency when pages are installed separately. WAL introduction
The new failure model is interrupted persistence. The new verification work includes crashes before append, during append, before flush, after flush, and before the client receives its reply.
Stage 3: make retries one logical operation
A client timeout can happen after commit. The engine stores an operation result in the same transaction as ownership changes. The key includes the authenticated actor and request identity, preventing unrelated users from accidentally sharing a deduplication namespace.
transfer(actor, request_id, item, expected_owner, new_owner):
if prior result exists for (actor, request_id):
require request parameters match recorded operation
return recorded result
require actor may transfer item
require current owner equals expected_owner
atomically change ownership and owner index
atomically record request parameters and resultMatching parameters prevents a caller from reusing one identity for a different operation and receiving a misleading old result. The record’s retention policy defines how long this protection lasts. An old request beyond retention must be rejected or resolved through another durable history; absence cannot safely be assumed to mean “never processed.”
This stage changes neither storage throughput nor geographic scalability. It solves an application-visible ambiguity created by unreliable communication. Its cost is retained metadata and a lookup per logical operation, which must be included in capacity estimates.
Stage 4: choose concurrency from the workload
The single-owner execution path can handle many connected clients, but its exclusive service time is finite. At a hypothetical 25 microseconds per transaction, 20,000 transactions/s consume 0.5 execution-seconds per second, or 50% of one resource before other work. A 5 ms transaction occupies as much time as 200 ordinary transactions.
The marketplace’s hottest item may already require sequential decisions. Parallel execution on that one item can add locks or retries without dividing the invariant. A short single-writer path is therefore a plausible starting point. Independent tenants, however, can execute on separate resources if their maintained indexes and metadata do not introduce hidden shared state.
A parallel engine might use MVCC with validation, lock-based protection, or partition ownership. The important requirement is proving serializable behavior where the contract needs it, including range checks and absent-key uniqueness tests. Protecting only written rows is insufficient when a transaction’s decision depends on a predicate.
This stage introduces concurrent physical access and reclamation hazards. Index nodes cannot be freed while readers still hold pointers. Page latches, atomic publication, and version cleanup become part of correctness. Choosing parallelism is not merely increasing a worker count.
Stage 5: separate execution from durability waiting
Synchronous persistence can leave an execution resource idle while storage flushes. Group commit allows several transactions to share a durability operation, subject to correct ordering and visibility rules. Bounded batching can improve throughput while adding a controlled waiting interval.
Suppose a flush takes 2 ms and a batch contains 40 commits. The flush duration divided by batch size is 50 microseconds per commit as an amortized resource measure. Each commit has not necessarily experienced 50 microseconds of latency: it may wait for batch formation and the full flush. Throughput amortization and request latency are distinct.
For a concrete safe branch, one logical executor applies transactions in order to projected state, including earlier uncommitted proposals; only the permitted committed visibility watermark is exposed to readers. Their read assumptions remain valid through that ordering. Acknowledgment waits for the required durable outcome. If leadership or recovery loses an uncommitted suffix, discard and rebuild its projected state from committed history; never commit stale precomputed write sets after their assumptions change. A parallel alternative must retain protection or validate dependencies through the outcome. The engine must bound in-flight state. If storage slows and the engine continues accepting private write sets indefinitely, memory becomes an unbounded queue.
For the marketplace, admission should reject excess attempts early with a definitive pre-mutation outcome. For the tenant service, fairness should stop one tenant filling all durability batches. For the shared world, transaction completion must not wait for a slow client’s socket to drain. One mechanism creates different scheduling requirements across workloads.
Stage 6: add checkpoints and bounded recovery
An ever-growing log makes restart increasingly expensive. A checkpoint records recoverable state corresponding to a known history position, with a publication protocol that ensures required log remains available until the checkpoint is valid.
A generic sequence writes checkpoint data under a new generation, verifies completion, durably publishes its manifest, and only then retires log segments made unnecessary by the new recovery point. Deleting the old log before publication can leave no complete recovery path after a crash.
Assume replayable log is 120 GB and effective replay throughput is 400 MB/s. Byte processing takes about 300 seconds using decimal units. A checkpoint reducing required replay to 12 GB changes that component to 30 seconds, but writing checkpoints consumes ongoing I/O. More frequent checkpoints trade background work for shorter recovery.
Backup and checkpoint are separate roles. A checkpoint on the same device does not protect against device loss or replicated logical deletion. An independently retained backup and log archive are necessary for historical recovery under that contract.
The new failure model includes interrupted checkpoint publication and missing recovery dependencies. Verification must show that every retained recovery point has a complete, decryptable chain of state and log.
Stage 7: replicate an authoritative history
Machine loss requires copies in other failure domains. A replicated log with a consensus protocol can establish one committed history despite leader changes under its stated assumptions. The reference design uses a Raft-like teaching model rather than inventing a new protocol.
A leader proposes a transaction’s deterministic write result, followers persist appropriate log entries, and the protocol determines commitment. Application to serving structures and client acknowledgment follow the chosen safety rules. A new leader must preserve committed history and cannot accept obsolete authority from a resumed former leader. Raft extended paper
client -> leader -> local log
-> follower log in domain B
-> follower log in domain C
committed history -> apply -> acknowledgeFigure: A replicated write path. Quorum and persistence details must follow the actual protocol; arrows show dependencies rather than measured durations.
The new failure model includes message loss, delay, partitions, pauses, and stale leaders. A timeout is suspicion, not proof that another leader is dead. Terms or epochs identify authority. External resources need fencing if an old process could continue acting outside the replicated state machine.
Three replicas do not triple write capacity if all execute or retain the same history. They purchase resilience and potentially read options, while adding network, storage, and recovery work.
Stage 8: define reads and failover capacity
Reads can go through current authority, use a protocol proving a replica’s safe position, or explicitly request an older snapshot. Each choice has a contract. A replica serving stale state cannot be called a current read merely because it is geographically close.
The marketplace needs current protected stock decisions, though product descriptions can be stale. The tenant service can offer read-your-writes sessions and historical reporting separately. The shared world can display recent cached positions while authoritative actions still validate current state. Separating these interfaces avoids paying for unnecessary freshness without weakening essential rules.
Capacity during failure is part of this design. Four interchangeable serving nodes at 70% occupancy represent 2.8 node-equivalents of demand. After losing one, the survivors average about 93.3% before rebuilding replicas. Even if correctness survives, tail latency may become unacceptable. Recovery traffic needs explicit capacity and scheduling.
Failover testing should measure time until useful service returns, not merely time until a leader is elected. Cache warming, log application, routing refresh, and reconnection can extend the outage experienced by users.
Stage 9: shard along real ownership boundaries
One replicated group still has finite capacity. Sharding creates independently executable units, each with authoritative routing and a movement protocol. A placement map needs versions so stale routers can be redirected safely.
For the tenant service, tenant identifiers are a natural boundary if most transactions stay inside one tenant. The marketplace can partition independent listings, but one popular listing remains an indivisible contention point unless stock rights can be safely subdivided. The world can partition rooms or regions, but crossing a boundary creates handoff and cross-region interaction problems.
A moving shard must account for writes during copying. A safe protocol establishes a consistent copy position, catches up changes, fences old ownership, and publishes the new owner under a durable routing transition. Copying files and changing DNS is insufficient when both locations can accept writes.
This stage introduces partial ownership movement. Failure between copying and publication must have an authoritative resolution. Rebalancing must also reserve bandwidth: moving hot state when every node is saturated can worsen the overload that triggered movement.
Stage 10: cross-shard operations force a choice
The reference engine cannot pretend a local transaction remains local after keys move to different shards. A transfer across owners needs an atomic distributed transaction, a business workflow with compensation, or a restriction requiring co-location.
For exact atomicity, participants and the coordinating decision must be recoverable. Replicating each participant does not replace transaction coordination. For a saga, intermediate states are visible and compensation may not restore the original world if external effects have occurred. The application must intentionally accept that model.
FoundationDB’s architectural paper is a useful comparison because it separates transactional processing and storage responsibilities while supporting a transactional key-value interface. Its organization shows one way to build a distributed substrate without making the application manage every participant itself. FoundationDB paper
The marketplace may require exact settlement across accounts. The tenant service can avoid most cross-tenant transactions by product design. The shared world may transfer a player through a durable handoff state that temporarily restricts actions rather than making every movement a global transaction. Different choices follow different user-visible contracts.
Stage 11: maintain realtime state without coupling clients to commit
The engine adds a change position to committed state and offers a snapshot aligned with that position. Clients apply a snapshot and then the required subsequent changes. Reconnect resumes from a retained position or obtains a new snapshot if history is unavailable.
Subscription authorization is server-enforced. Shared query computation may reduce evaluation work, but delivery remains per recipient. Transaction batches preserve coherent client transitions where the API promises them. Slow consumers receive bounded buffers and a resynchronization policy rather than holding database locks while their network drains.
For 20,000 changes/s, 100 recipients/change, and 80 payload bytes, output is 160 MB/s before overhead. Reducing recipients to 20 yields 32 MB/s. This optimization changes network and delivery work, not transaction execution speed.
SpacetimeDB illustrates the integration of authoritative execution and subscribed state, while its scaling explanation explicitly separates computation, storage, and networking limits. The reference architecture borrows that analytical separation without assuming every integrated engine implements the same mechanisms. SpacetimeDB scaling discussion
The shared world’s simulation layer still owns prediction, interpolation, reconciliation, and any lag compensation. A database subscription supplies state, not a complete motion protocol.
Stage 12: operate under overload rather than assuming it away
Admission control distinguishes offered, admitted, and completed work. A bounded queue prevents latency and memory from growing without limit. Per-tenant budgets account for estimated CPU, I/O, retained state, and response size rather than treating every request as equal.
Retries have deadlines and budgets. A definitive pre-mutation rejection differs from an uncertain timeout after potential commit. Backoff reduces synchronized pressure, but it does not create capacity. The engine reserves resources for log cleanup, replica repair, checkpointing, and control operations so overload does not prevent recovery.
If 12,000 attempts/s arrive while safe completion capacity is 10,000/s, backlog grows by 2,000/s absent rejection. A 20,000-request queue fills in ten seconds. Adding a retry from every timed-out request can increase offered load further. The proper response is to protect useful work, not keep enlarging the queue.
The marketplace can quickly return sold-out and deduplicate repeated attempts. The tenant service can isolate expensive scans. The world can reduce optional update frequency while retaining exact ownership transitions. Degradation must respect business invariants and be explicit in the product contract.
Stage 13: make operational state part of the design
Metrics connect request traces to queues, storage flushes, conflict rates, log lag, subscription delivery, and retained versions. Logs avoid secrets and carry operation identity for uncertain-outcome investigation. Authorization checks apply to queries, subscriptions, backups, and administrative tools.
Schema evolution uses compatible stages. New readers tolerate old rows, old writers remain supported during transition, backfills avoid overwriting concurrent updates, and constraints are validated before old representations disappear. Rollback is possible only while the old code can interpret the resulting data.
Recovery procedures specify RPO and RTO for machine loss, regional loss, and logical corruption separately. A replicated accidental deletion requires historical recovery rather than ordinary failover. Backup restoration is tested with the actual encryption keys and dependencies needed in a disaster.
The new failure model is human and operational: incorrect deployment, bad migration, leaked credentials, corrupted retained state, and unavailable key services. These are ordinary engineering concerns, not rare exceptions outside the database architecture.
Three final architectures, three reasons
For the contended marketplace, keep exact inventory transitions short and colocated where possible. Use durable request identities, bounded admission, and a strong settlement path. Serve descriptions and analytics through explicitly derived representations. Parallelism helps independent listings; it does not divide one final item into multiple owners.
For the multitenant service, align data and transaction boundaries with tenants, share capacity with measured fairness, and split unusually large tenants when their access patterns permit it. Avoid unnecessary global counters and indexes that turn independent work into coordination. Backfill, backup, and noisy-neighbor behavior matter as much as healthy-state throughput.
For the shared world, integrate authoritative actions with selective state delivery, partition by interaction locality, and define durable handoffs. Budget fan-out and client backpressure separately from state execution. Keep persistent ownership distinct from presentation prediction. One room can be sequential and fast while many independent rooms use parallel capacity.
These architectures share mechanisms but differ in placement, scheduling, and interface contracts. That is the final result of the course: scalability is the ability to divide the actual work while preserving the invariants that make the application useful.
A final failure trace through the complete system
Take one marketplace purchase after all stages are present. The client sends request X, the gateway authenticates it, and admission grants a bounded execution slot. Routing identifies the current listing owner. The transaction validates stock and identity, changes ownership, and records X’s result. Its log history becomes committed under the replication policy. Only then does the acknowledgment path claim the corresponding success contract.
Now fail the gateway before the reply reaches the client. The operation remains committed. A retry through another gateway carries X and receives the original outcome. A subscription may independently deliver the new ownership, but the application does not use receipt of one animation event as its only proof of purchase. These interfaces agree because they derive from the same committed state, with explicitly different delivery timing.
Next, let the old shard owner resume after a leadership change. Its obsolete epoch cannot authorize new writes. If it tries to publish a stale external effect, the receiving resource must reject the old fencing token or the workflow must otherwise ensure idempotency. Consensus inside the database does not automatically control an unrelated external service.
Finally, suppose a bad migration later removes ownership records. Replicas reproduce the deletion correctly. Recovery uses an independent historical backup and log chain, restores into isolation, validates the requested point, and reconciles legitimate operations after that point under an explicit business procedure. Ordinary failover would not restore the old data.
This trace crosses admission, routing, transactions, logging, consensus, retry identity, subscriptions, fencing, and recovery. Each mechanism closes a different gap. Removing one because another also sounds like “reliability” leaves a specific failure unhandled.
What the architecture has actually optimized
The reference design avoids full scans with indexes, avoids repeated process crossings by locating computation appropriately, amortizes persistence where the latency contract permits it, and parallelizes independent ownership domains. It avoids repeated query work through maintained results, avoids unnecessary delivery through interest selection, and avoids unlimited waiting through admission control.
It has also accepted costs: indexes consume writes, logs consume storage bandwidth, replicas consume network and devices, versions consume retention space, subscriptions consume evaluation and delivery resources, and backups consume independent capacity. The architecture is scalable when added resources increase useful capacity while per-operation coordination and maintenance remain bounded under the stated workload and correctness contract.
That is the standard against which both a new engine and a marketed platform should be evaluated. Identify the removed work, the retained obligations, and the conditions that move the bottleneck. Low latency and high concurrency are outcomes of those choices rather than isolated features that can be added at the end.