A new worker starts with an empty cache

Our engine has used memory to keep useful pages close to its operators. Now put durable data in a storage service that several query workers can access. A worker performs computation; the storage service owns the persistent data.

This separation can let us add workers without copying the entire database onto each worker first. It does not make the data immediately available in every worker’s memory.

Follow the first query on a new worker

Worker A has already cached the shop’s popular product pages. A new worker B has an empty cache. The same query that A answers from memory forces B to fetch pages across the network.

Warm worker A

Query → local cached pages → result
Few remote reads for this workload.

New worker B

Query → remote storage → cache → result
Many remote reads until useful pages accumulate.

The workers run the same operators. Their different cache contents produce different demands on shared storage.

Suppose shared storage can sustainably serve 100 MB each second for this workload. Existing workers already consume 70 MB/s. If B tries to warm its cache at 50 MB/s, total demand becomes 120 MB/s. Without another change, requests must queue or receive less throughput than they request.

Adding CPU has exposed a storage limit. Even A can slow down because its remaining cache misses now compete with B’s warm-up traffic.

Treat readiness as measured capacity

A simple policy limits B’s warm-up reads to 20 MB/s while measuring the effect on existing requests. That leaves 10 MB/s of the assumed storage capacity as headroom, though real latency and changing demand must still be checked.

If B needs 6,000 MB of useful pages and can fetch them at a steady 20 MB/s, the transfer alone takes 300 seconds, or five minutes. This is an illustrative lower-bound calculation: duplicated reads, evictions, processing, and foreground work can make readiness slower.

The design also needs metadata telling workers which stored objects are current. Adding multiple workers does not automatically coordinate transactions, ownership, or publication of new data. Those remain explicit responsibilities. Nor does remote storage alone guarantee independent failures if all workers depend on one unavailable service.

Predict the second new worker

B and C both warm at 20 MB/s while existing demand remains 70 MB/s. Does the earlier 10 MB/s headroom remain?

Add the shared demand

No. Total demand becomes 110 MB/s, exceeding the assumed 100 MB/s capacity. A per-worker limit is insufficient if it ignores how many workers are warming. Admission and scheduling need to account for the shared total.

Separating resources gives more placement choices. It also makes network reads, cache readiness, and shared bottlenecks visible parts of the engine’s behavior.

Historical example: the 2017 Amazon Aurora architecture paper. The worker model and numbers here are illustrative, not Aurora performance measurements.

Extended reference

The extended chapter preserves the original technical coverage, additional examples, and citations. The explanation above supplies the concepts needed for this page; the reference develops the specialist cases.

Find a section from the original chapter