Hot keys and geographic stock

Start with the guided lesson. These sections develop the mechanisms and tradeoffs beyond that first example.

Uniform traffic and hot traffic are different problems

A uniform access distribution spreads requests approximately evenly across keys. It often makes partitioned systems look attractive: independent partitions perform independent work. Real workloads commonly contain skew, in which a minority of keys receive a disproportionate fraction of requests. A popular auction, shared counter, tenant, or world region can dominate the critical path.

For an illustrative arrival rate of 100,000 requests per second, suppose 60% target one inventory key. That key receives 60,000 requests per second. The remaining 40,000 may be easily distributed across thousands of partitions. The average rate per partition can therefore look low even while the single hot partition is saturated. Adding partitions without changing ownership of the hot invariant does not divide the hot work.

Contention means operations compete for something that cannot be used independently under the required contract. The shared resource can be a logical record, an index page, a lock table, a cache line, a storage device, or a network connection. Logical independence does not guarantee physical independence. Two unrelated rows might share one index page, while two independent counters might repeatedly transfer ownership of the same cache line between CPU cores.

A hot record does not imply that every operation must perform a long critical section. A guarded update might execute quickly inside the engine. An application that holds a lock while making several remote calls can make the same logical operation far slower. Later lessons distinguish unavoidable serialization of an invariant from avoidable waiting introduced by implementation boundaries.

Compare architectures while holding the contract fixed

A fair architectural comparison starts with the same invariant, acknowledgment boundary, request mix, and data distribution. Consider a purchase implemented in an application server and a relational database. The server might perform one guarded statement plus an order insertion inside a transaction. Another implementation might perform several read-modify-write exchanges. These are materially different implementations even though both use the same database. Comparing the latter against an integrated stored-computation engine primarily demonstrates a boundary and transaction-design difference.

Likewise, a system acknowledging mutations before persistence is being asked a different question from one acknowledging after a durable replicated commit. Both can be useful, but their failure contracts must appear next to the latency numbers. A comparison with ten subscribers per change does not predict behavior with ten thousand subscribers. A small dataset fitting in RAM does not expose storage and recovery behavior for a dataset twenty times larger.

A compact decision record can therefore state: the invariant; the maximum acceptable age of reads; the durability point; the hottest-key rate; the data and index sizes; the required latency percentile; and behavior during overload or failure. The record becomes a stable reference as implementation choices change. It is particularly useful when evaluating systems that combine functions, database state, and client synchronization, because apparently removed work may have moved into subscription maintenance or background recovery.

Scenario: reserving inventory without making correctness approximate

Suppose a global retailer owns 12,000 units and allocates 4,000 units to each of three regions. A region may sell only from its current allocation. Purchases can then avoid contacting another region for every unit, provided the allocation protocol guarantees that the sum of regional rights never exceeds total stock. The global stock invariant is preserved by distributing authority before the customer request, rather than by letting each region independently guess global availability.

The tradeoff is visible when one region exhausts its 4,000 units while another still has 3,000. Customers can see “unavailable here” despite remaining global inventory until rights are transferred. Moving rights requires its own safe state transition; duplicating a transfer message must not duplicate selling rights. Geographic placement improves local latency only because the design makes ownership explicit. It has not eliminated coordination, but changed when and how often coordination occurs.

This scenario provides a recurring test for later mechanisms. Locks can protect one allocation locally. Logs can preserve committed allocation changes. Replication can keep regional authority available after machine failures. Distributed transactions or carefully designed transfer protocols can move rights across regions. Each mechanism answers one part of the problem, and the workload contract prevents those answers from being confused.

Read the workload at both scales

The table labels attempts, not commits. Acknowledgment follows the authoritative decision under the selected durability contract; client visibility can occur later through response delivery or a subscribed update.

Sources and further reading

• PostgreSQL 18 transaction isolation
• CMU database systems curriculum