Separated compute and storage — extended reference

This reference preserves the earlier detailed chapter and its citations. Start with the guided explanation, then use these sections for additional mechanisms and qualifications. References to the original lesson sequence are historical.

Part 6: Scaling execution and state delivery

Prerequisites: Lesson 35 and the preceding foundations.

Learning outcomes

Compare shared-nothing, shared-storage, and separated compute/storage architectures.
Explain remote-read latency, object-storage granularity, durability ownership, and cache warming.
Evaluate elasticity using startup, placement, cache state, and recovery costs as well as steady-state throughput.

Elastic compute starts with cold state

A service adds a new compute node expecting an immediate reduction in latency. The node’s caches are empty. It reads aggressively from remote storage, increasing shared network and storage load. For several minutes the old nodes become slower too. More compute existed, but useful warm capacity had not yet arrived.

Separating compute and storage can improve elasticity and resource utilization. It also introduces remote access, metadata, cache ownership, and failure coordination. The design must identify which responsibilities are actually independent and which remain connected through a shared path.

Shared-nothing and shared-storage organizations

In a shared-nothing organization, each node owns some data and local execution resources. Scaling and recovery often move or replicate partitions. Local access can be efficient, while cross-partition operations and rebalancing create communication.

In a shared-storage organization, several compute nodes can access a common storage service under a defined concurrency and consistency protocol. This can reduce data movement when changing compute membership. It does not mean every node can independently overwrite pages without coordination.

A storage service can itself be distributed and replicated. The apparent shared layer still has ownership, consistency, bandwidth, and failure domains internally. The architecture should trace through that abstraction rather than treating it as an infinitely fast durable file.

Separate data, log, metadata, and cache

A system may place the durable log in one service, data pages in another, metadata in a coordinator, and hot state in compute-local caches. These paths have different access patterns and failure requirements.

The log often favors ordered append and durable acknowledgment. Data storage favors efficient reads and background reconstruction or merging. Metadata determines authoritative mappings and versions. Caches reduce remote work but are normally reconstructible under the design’s rules.

A diagram should identify the critical write path separately from background page materialization:

compute -> durable log service -> acknowledgment boundary
   |
   +-> local cache/current state

background storage work -> persistent page representation

This is a generic schematic, not a description of every disaggregated product. The exact dependency ordering must be established by the engine’s recovery protocol. Acknowledging from reconstructible log state can be safe only if the log and metadata together preserve the required outcome.

Remote reads and caching

A remote page miss adds network transit, service queueing, and storage work. Batching and prefetch can improve throughput, while point-dependent access remains sensitive to latency. A cache can hide most remote reads under a warm workload and expose them sharply after failover.

For an illustrative 2 TB working set loaded at an effective 1 GB/s, a full warm-up requires 2,000 seconds, about 33.3 minutes. This assumes decimal units and ignores overlap, selective warming, contention, and reuse. The calculation demonstrates why provisioning time is not the same as usable-capacity time.

Cache placement and coherence depend on write ownership. Immutable versioned pages can be shared more easily than mutable pages with several independent writers. Version identifiers help reject stale cached objects, but readers still need to know which version is valid for their snapshot.

Object storage and tiering

Object storage can offer economical durable capacity and useful scaling properties, with access granularity and latency different from local memory or SSD. An engine using it must design around object creation, metadata publication, retries, and the relevant consistency semantics.

Tiering keeps hot data near compute and colder data in cheaper storage. A tiering policy estimates future reuse and moves data in the background. Misclassification creates expensive misses; moving data consumes bandwidth and can interfere with current traffic.

Large objects can make sequential scans efficient while increasing amplification for small random reads. Small objects improve granularity while increasing request and metadata overhead. Compression, block indexes, and local caches help trade these costs, but each adds a maintained representation.

Elasticity and failure boundaries

Independent compute scaling is valuable when the durable state does not need to be copied to every new worker before useful execution. Even then, workers need metadata, credentials, compatible code, and enough cached state to meet performance goals.

A shared storage outage can affect many compute nodes simultaneously. Separating services changes the failure domain; it does not remove correlated failure. The compute layer must handle remote latency and unavailability without acknowledging work that the durability protocol cannot preserve.

Retries at several layers can amplify a storage incident. Compute retries, storage retries, and client retries need coordinated budgets. A storage timeout after a durable append can also create uncertain outcomes, requiring the same operation-identity reasoning as other remote calls.

Economics and resource placement

Disaggregation can avoid paying for storage and compute in fixed proportions. A large cold dataset with modest active computation can benefit. A tightly coupled low-latency workload may instead pay substantial network and remote-access costs.

Cost includes requests, transfer, retained log, replicas, cache memory, and background maintenance, not just storage bytes. Scaling compute down can reduce immediate expense while increasing later cold-start work. Retaining a small warm pool can trade fixed cost for predictable latency.

A comparison should preserve the service objective. A cheaper configuration that takes half an hour to regain performance after scaling may be suitable for batch analytics and unsuitable for an interactive marketplace. The architecture’s elasticity must be defined in terms of useful workload behavior.

Historical case connections

The Aurora paper describes design choices for a cloud relational system with a specialized distributed storage layer. The Snowflake paper describes a multi-cluster shared-data analytical architecture. Their historical mechanisms illustrate different motivations for separating responsibilities; neither paper should be treated as a complete statement of every current service feature.

The analysis is that both designs exploit different scaling units while paying for coordination and data access at their chosen boundaries. Their workload priorities differ. A transaction log path and an analytical scan path have different latency, durability, and batching needs even when both use remote storage.

Follow one write across the separated paths

Consider a hypothetical database whose compute process evaluates transactions, a log service durably orders accepted write sets, and a page service materializes data from that history. The design is illustrative; actual products choose different divisions.

A transaction reads a coherent version, validates its assumptions under the concurrency protocol, constructs mutations, and submits them with a stable identity. The log service establishes the required durable outcome. Serving state advances only under the defined visibility rules. Background page construction can lag if recovery and reads know how to combine page state with the retained log suffix.

transaction evaluation
   -> protected / validated write set
   -> durable ordered log decision
   -> permitted visibility watermark
   -> background persistent page generation

Figure: A generic dependency chain. Separating services does not permit arbitrary reordering of validation, commit, and visibility.

If the page service has materialized only through position 900 while a read snapshot requires 950, the system needs the missing changes or must wait. Reading the older page without reconstruction returns stale state. A page identifier should therefore encode or accompany enough version information for the reader to know what history it represents.

A network timeout during log submission is an uncertain outcome. The compute process cannot safely resubmit an increment as a new transaction without resolving identity. Moving the log to a service creates the same retry ambiguity as any remote mutation, now inside the engine.

Immutable data objects can be cached broadly because their bytes do not change after publication. A new version receives a new identity, and metadata points readers to the appropriate version. This avoids one class of cache invalidation, but shifts correctness into metadata publication and retention.

Suppose a compactor writes object C from objects A and B. It must finish and verify C before publishing a manifest that references it. Only after no supported reader or recovery path needs A and B can they be deleted. A crash after writing C but before publication leaves an orphan that can later be collected; a crash after deleting A and B but before durable publication can leave no recoverable state.

Content checksums detect selected corruption and allow validation after transfer. They do not decide whether the object belongs to the current database history. A perfectly intact object from an obsolete generation remains the wrong object for a current read.

Garbage collection is therefore a distributed protocol. It depends on retained snapshots, backups, replication state, and readers that may still reference old generations. Aggressively deleting “unreferenced” files from one process’s current view can violate another process’s valid historical read.

Remote access changes the value of prefetch

A dependent B-tree traversal can require several remote misses, with each child address known only after reading its parent. If three misses each cost an illustrative 2 ms, the dependent component is about 6 ms. Fetching three independent known blocks in parallel can approach 2 ms plus overhead instead. Dependency shape matters as much as bytes.

Caching upper index levels can remove serial remote steps for point lookups. Prefetch can hide known future access for scans. Neither helps an unpredictable pointer chase equally. Layout choices that increase locality can therefore be more important once the storage boundary includes a network service.

Read amplification also changes economics. Fetching a 4 MB object to use one 4 KB block is a thousandfold byte ratio in the idealized example. Range reads, local block caches, and object indexing can reduce transferred bytes while adding requests and metadata. The best object size balances throughput, request cost, reuse, and latency.

A remote service’s peak bandwidth does not guarantee small-read latency. Queueing from large scans can delay point misses. Separate priority classes or caches can help, while shared backend resources remain a coupling point.

Cold caches are a placement event

A newly started compute process may have a valid view of metadata yet no useful cached blocks. If it immediately receives a full traffic share, latency rises and remote read demand surges. Existing nodes can become slower if they share the same storage service.

Suppose a workload needs a 200 GB hot set and the new node can safely warm at 400 MB/s. Full transfer takes at least 500 seconds, about 8.3 minutes. Selective warming can reduce the required bytes, but identifying the hot set has its own assumptions. A previous owner’s access profile can be useful until the workload changes.

A controlled ramp increases traffic as hit rate and response time become acceptable. The node’s readiness condition should reflect useful workload capacity rather than merely successful process startup. Sending one health-check query does not establish a warm working set.

Failover can be worse than routine scale-out because the old warm owner is unavailable. A design can retain recoverable caches, keep warm standbys, or accept a documented degraded interval. Each trades steady cost against recovery latency.

Compute scaling does not eliminate transaction authority

Several compute workers reading shared storage need a coordination model for conflicting writes. They might own disjoint partitions, route writes through one authority, or use a transaction service. Shared access to the same durable bytes does not authorize independent overwrites.

A useful counterexample is two workers reading stock 1 from the same object and each writing stock 0 plus a different purchase. Object durability preserves both writes only according to the storage interface, but the business invariant can still fail. Atomic object replacement is not automatically an atomic multi-record transaction.

The engine must define compare-and-swap semantics, version validation, locking, or another protocol at the relevant scope. Storage consistency and database serializability operate at different abstraction levels. A storage service can be correctly strongly consistent for individual object operations while the database layer implements an invalid transaction.

The inverse distinction also matters. A sophisticated transaction service cannot make acknowledged data survive if its required storage copies were never made durable. The full proof crosses both layers.

Failure independence requires actual separation

Compute and storage can be separate services while sharing a region, network fabric, identity provider, or control plane. An outage in one shared dependency can affect both. Counting service names is not counting independent failure domains.

A storage timeout can arise from queueing, packet loss, access denial, or an unavailable replica quorum. Blindly retrying from every compute node can multiply demand and prolong the outage. Retry budgets should account for aggregate callers and preserve operation identity on writes.

The control plane matters as well. If existing data nodes can serve requests during a metadata-service outage, the system should state which operations remain safe. New allocations, resharding, or failover may still require unavailable control services. “Data plane is up” is not equivalent to unrestricted availability.

Credentials and encryption keys can create another shared dependency. A cache may keep a running node functional briefly while a newly started node cannot retrieve a key. Disaster recovery must test these paths rather than assuming all supporting services are available whenever storage bytes survive.

A cost model beyond dollars per gigabyte

Assume a hypothetical analytical service retains 100 TB but actively scans only 1 TB each day. Cheap object storage can align cost with cold capacity, while compute scales for scheduled reporting. A low-latency transactional service touching unpredictable small records may incur many remote requests and need substantial cache memory, changing the economics.

A cost model includes stored bytes, replicated copies, read requests, write requests, transfer, log retention, compute time, cache memory, and maintenance. It should include transition costs such as warming after scale-down. Scaling to zero can save idle compute and make the next request pay initialization and cold reads.

Suppose keeping a warm worker costs C per hour, and restarting it requires W units of transfer and computation each time. Frequent short idle periods can cost more in repeated warm-up than maintaining the worker. The crossover depends on actual prices and workload; the mechanism can be reasoned about without inventing current vendor pricing.

Operational complexity is also a cost. Debugging a latency spike across compute, log, metadata, and page services requires correlated observability. Independent scaling is valuable when it solves an important resource mismatch, not merely because the architecture diagram has more independently deployable boxes.

Historical comparison: two motivations for separation

The Aurora paper presents a cloud transactional design in which a specialized distributed storage service changes how logging and page work are performed. The Snowflake paper presents an analytical architecture with shared durable data and independently provisioned compute clusters. These are historical publications with different workload goals, not interchangeable implementation recipes. Aurora paper, Snowflake paper

The original analysis here is that separation can optimize different imbalances. Transactional systems may seek efficient durable commit and recovery while preserving short updates. Analytical systems may seek independent compute groups reading shared history without duplicating all persistent data. Both pay remote-access and metadata costs, but those costs occur in different critical paths.

A comparison should therefore ask which scaling unit is being changed: query workers, transaction authority, durable log capacity, data storage, or cache. “Compute and storage scale independently” can be true for one resource pair while a shared transaction coordinator remains a bottleneck.

A worked elasticity scenario

A reporting service runs on two warm compute groups. A third is started for a large customer export. The export reads 500 GB, while ordinary reports repeatedly access a 50 GB hot set. If all groups share one remote bandwidth limit, the export can evict useful cache locally and consume shared bandwidth remotely.

A separate compute group protects CPU scheduling but does not automatically protect storage. Rate-limiting the export, prepositioning data, or giving it a separate storage budget can preserve ordinary reports. The appropriate choice depends on completion deadlines and whether duplicate cached data is affordable.

Now switch to a transactional scenario. A new compute node receives writes but still consults one shared authoritative log path. CPU headroom improves, but commit throughput remains unchanged because the log path is saturated. Adding compute cannot solve that ceiling. Group commit, log partitioning where invariants permit it, or a different durability organization addresses the actual constrained resource.

The scenarios demonstrate why scaling claims should be phrased in terms of useful capacity. A resource can scale independently while the operation’s end-to-end path remains constrained elsewhere.

Reference checklist expressed as a design argument

A complete separated architecture can explain where authoritative versions live, where a write becomes durable, how reads choose a version, and which cached objects may be discarded. It can demonstrate safe metadata publication and identify when old objects become reclaimable. It can quantify remote misses and explain cold-start behavior.

It also states which service failures stop reads, writes, membership changes, and recovery. It accounts for shared dependencies and reserves bandwidth for rebuilding. Its economics include request and transition costs rather than only bytes at rest.

The point of this argument is not to make every design elaborate. A local engine may remain the smallest correct solution for a bounded workload. Separation becomes justified when independent resource scaling, workload isolation, or recovery behavior creates benefits larger than the new communication and coordination costs.

An elasticity timeline with a readiness boundary

start compute -> obtain metadata/keys -> validate serving version
             -> warm required hot blocks -> ramp admitted traffic
             -> reach tested useful capacity

Figure: A generic compute replacement timeline. Process startup precedes readiness to meet the workload’s latency objective.

Suppose startup and metadata take 20 seconds, targeted warming takes 180 seconds, and traffic ramps over another 60 seconds. The useful-capacity point is roughly 260 seconds if these stages are sequential. Some warming can overlap the ramp, but that overlap increases remote demand and must be measured. Reporting only the 20-second startup gives an incomplete elasticity claim.

The readiness check should be tied to the intended workload. A point-lookup probe may pass while a large analytical scan remains cold. A transaction probe may pass while its required replica path is still degraded. Different workload classes can become ready at different times.

Scale-in should preserve enough warm and recoverable state for the next expected burst. A controller can reduce cost by removing truly idle capacity while retaining a minimum warm footprint. That is an economic and latency tradeoff rather than a failure of elasticity.

A separated architecture is most persuasive when it explains these transitions in the same detail as its steady-state diagram. It must show who owns durability, how remote misses are served, where caches are valid, and what temporary resources replacement consumes. Without those details, independent scaling is a topology description rather than a demonstrated service capability.

One final dependency deserves explicit naming: storage metadata must remain reachable and trustworthy when compute changes. Durable objects without the manifest identifying their valid versions do not form a usable database. Metadata backup, authority, and recovery belong in the separated design alongside the larger data objects, even when their byte volume is small.

Design conclusion

A disaggregated design should map each byte and decision to an owner: log, data, metadata, cache, and execution. It should quantify remote misses, warm-up, transition work, failure coupling, and the cost model relevant to the workload.

The next part places database mechanisms inside an application and operational system. An engine can supply strong primitives while an application still creates unsafe transaction boundaries, duplicated side effects, or unbounded request patterns around them.

Sources and further reading

Course index · Lesson plan