Capacity and multitenancy — extended reference

This reference preserves the earlier detailed chapter and its citations. Start with the guided explanation, then use these sections for additional mechanisms and qualifications. References to the original lesson sequence are historical.

Part 6: Scaling execution and state delivery

Prerequisites: Lesson 34 and the preceding foundations.

Learning outcomes

• Convert workload demand into CPU, memory, storage, and network budgets.
• Explain hot keys, tenant quotas, noisy neighbors, and failover headroom.
• Evaluate autoscaling against a particular bottleneck and distinguish isolation from average utilization.

A spare CPU does not imply spare database capacity

One tenant scans a large table while another performs small durable updates. CPU utilization appears moderate, but the storage queue grows and transactional latency deteriorates. An autoscaler adds CPU capacity without increasing storage bandwidth. The symptom persists because the constrained resource was misidentified.

Capacity is a vector: CPU execution, memory residency and retention, device bandwidth and operations, network throughput, synchronization, and sometimes authoritative metadata throughput. A multitenant design also needs isolation so one tenant cannot consume every dimension needed by others.

Per-operation demand

Start with operation classes and their resource demands. A read can use little CPU but miss storage. A write can generate log, index, replication, and compaction work. A subscription can be cheap to evaluate but expensive to deliver to many recipients.

For illustrative 80,000 operations per second at 40 microseconds CPU each, demand is 80000 × 0.000040 = 3.2 CPU-seconds/second. This is fully utilized equivalent execution demand before scheduling and other work. It does not mean four cores automatically meet a tail-latency objective.

If each operation also creates 2 KB of network payload, the payload rate is 160 MB/s using decimal units. Three replicas and subscribers can add other streams. The capacity model should avoid counting a byte twice while still identifying each actual transfer boundary.

Memory as capacity and retention

Memory holds data caches, queries, versions, connections, and backlogs. A per-request allocation multiplied by concurrency can dominate. A 100 KB retained request state with 100,000 in-flight requests uses 10 GB before allocator overhead.

Long snapshots can retain versions independent of current CPU load. Slow subscribers can retain output or log history. A memory leak and a valid but unbounded retention policy can look similar operationally while requiring different fixes. The engine needs visibility into ownership and age, not just total resident bytes.

A capacity forecast should distinguish hot working-set growth from total historical data growth. Adding storage can accommodate history without keeping hot queries fast if the active set no longer fits memory. Conversely, a large archive may have little effect on a small well-isolated hot partition.

Storage bandwidth and maintenance debt

Foreground writes create background work. LSM compaction, index maintenance, checkpoints, vacuum, backup, and repair compete for device resources. Sustainable capacity includes enough budget to retire that work under realistic retention and failure conditions.

RocksDB’s write-stall documentation illustrates that limiting ingestion can be a response to maintenance pressure. The architectural interpretation is that an unlimited foreground rate cannot exceed the system’s long-term ability to maintain its representation. Disabling the limit may exchange visible stalls for larger debt and eventual exhaustion.

A device’s advertised peak bandwidth is not a mixed-workload latency guarantee. Small random writes, reads, synchronization, and concurrent background transfers interact. Measure the relevant access pattern at the intended queue depth and reserve headroom for variability and recovery.

Tenant isolation

A tenant boundary can govern CPU time, memory, I/O, connections, query complexity, result size, and retained history. Limiting only requests per second ignores the difference between one point lookup and one billion-row scan.

Per-tenant resource accounting can estimate cost before admission and adjust after execution. Estimation errors need safe handling. A query should not allocate arbitrarily because its initial estimate was small. Hard limits can protect the service while soft budgets permit temporary borrowing.

Strict partitioning gives predictable isolation but can waste unused reservations. Shared pools improve utilization while allowing noisy neighbors. Weighted fairness, minimum guarantees, and controlled borrowing occupy a middle ground. The implementation must define how borrowed capacity is reclaimed without causing starvation or abrupt failure.

Hot keys and uneven work

Aggregate averages conceal hotspots. A tenant receiving 40% of writes can dominate one partition. A popular key can saturate an authoritative update path even after the surrounding partition is split. An index root, allocator, or global timestamp counter can become a physical hotspot across independent logical keys.

Diagnosis should distinguish data skew, request skew, and shared implementation metadata. Splitting data helps the first two only when it actually divides work. Sharding a logical counter changes its consistency or requires a combining protocol. Padding a cache line helps false sharing but does not divide a real invariant.

A useful load model includes heavy hitters and burst shape. Uniform synthetic keys can produce a misleading scalability curve. A workload matrix should vary skew and transaction duration independently to reveal the regime in which coordination dominates.

Autoscaling and useful capacity

Scaling decisions need lead time. Provisioning a node is only the first step. Data transfer, replica catch-up, cache warm-up, compaction state, and routing changes determine when it can serve useful work.

A cold node can increase demand on remote storage before it improves throughput. Moving a hot partition can temporarily duplicate I/O and network load. A safe scaling policy begins before the cluster has no spare capacity for the transition.

Scale-down also has costs. Removing a node can force movement, reduce failure tolerance, and evict useful caches. An idle period may be too short to repay the next warm-up. Hysteresis and minimum residency can prevent repeated churn.

Planning for a failure state

A healthy cluster’s capacity is not its degraded capacity. Losing one node can redistribute its traffic and trigger recovery simultaneously. If every remaining node was already near saturation, failover can preserve correctness while making the service unusably slow.

An illustrative four-node service using 70% of each node’s relevant capacity has 2.8 node-equivalents of demand. After losing one node, the same demand averages about 93.3% of the three survivors before recovery work. That leaves little slack for tail latency or catch-up. This arithmetic assumes equal interchangeable capacity and should be refined for real placement and skew.

Headroom is therefore part of the availability design. It is not always economically optimal to reserve for every extreme event, but the chosen service objective must acknowledge the tradeoff. A cost-efficient architecture makes that decision explicit rather than assuming healthy-state benchmarks apply during failure.

Constructing a resource-demand matrix

A resource vector translates business traffic into engine work. Consider three hypothetical classes: point reads, durable writes, and reports. A point read costs 20 microseconds CPU, 500 bytes of outgoing payload, and a storage miss with probability 0.01. A write costs 60 microseconds CPU, 2 KB of log, and 1 KB of replication payload per additional copy. A report costs 200 ms CPU and scans 100 MB. These coefficients are illustrative and must be measured for a real system.

At 50,000 reads/s, 10,000 writes/s, and two reports/s, foreground CPU demand is 1 + 0.6 + 0.4 = 2 CPU-seconds/s. Outgoing read payload is 25 MB/s. Log production is 20 MB/s. Report scan demand is 200 MB/s before cache effects. Five hundred read misses/s can add device operations even when their byte volume is small.

These columns cannot simply be added together because they describe different resources. They produce separate utilization estimates against available capacity. The limiting dimension determines which optimization matters. Reducing CPU by 20% does not solve an overloaded report-storage path, while caching reports may reduce storage but increase memory retention.

A vector also exposes amplification. With two additional replicas, each write’s 1 KB replication payload creates 20 MB/s of replication output from the assumed writer. If each replica independently performs maintenance, cluster device work grows again. Include real transfers and maintained copies without counting the same local byte twice.

Capacity coefficients are distributions

An average 20-microsecond read can hide a mixture of cheap hits and expensive misses. A report’s cost can vary with selectivity, skew, and result size. A capacity model should record distributions or meaningful classes rather than one universal request coefficient.

Suppose 90% of a tenant’s reads touch a small hot set and 10% scan a cold range. The hot set may fit comfortably in memory, while the cold reads dominate device work. Combining them into a mean can obscure why a small change in cold-read fraction dramatically affects latency.

Coefficients also change with scale. A larger index can add a level, a join can cross its spill threshold, and a working set can stop fitting in cache. Linear extrapolation is most reliable within a known operating regime. The model should mark thresholds at which the representation or execution path changes.

A useful forecast therefore combines measured coefficients, workload-shape assumptions, and sensitivity analysis. It can state, for example, that the service remains within device bandwidth if report frequency stays below a given rate and cache misses stay within a tested band. That is more actionable than an unsupported single maximum operations-per-second number.

Headroom must cover variance and transitions

Imagine four nodes, each with effective safe capacity of 10,000 work units/s under the target latency objective. Healthy demand is 24,000 units/s, averaging 60% of nominal total. After one node fails, survivors average 80% before repair. If repair consumes 10% of each survivor’s relevant capacity, only 27,000 units/s remain for user work, leaving a narrower margin.

The calculation assumes equal units and balanced placement. A hot tenant or a shard that cannot move freely can make one node exceed the average. Failure planning must therefore check placement-specific states, not only total remaining capacity.

Not all headroom needs to be idle hardware. Optional reports can pause, background exports can slow, and nonessential updates can be reduced under a documented degraded mode. Essential maintenance cannot be deferred indefinitely. A policy must identify how much work can safely stop and for how long.

A recovery budget includes network, I/O, CPU, and retained history. If a failed replica misses 100 GB and catch-up can use 200 MB/s without harming foreground service, transfer alone needs 500 seconds. A system promising redundancy restoration in one minute would require more bandwidth, less missing data, or a different recovery plan.

Resource ownership makes memory diagnosable

Total resident memory answers how much the process uses. It does not explain who can release it. A useful accounting system associates bytes with owners: buffer cache, versions, active queries, private write sets, connection buffers, subscription results, and background tasks.

A growing buffer cache can be bounded and evicted. Growing old versions may be held by one long snapshot. Growing output buffers may indicate slow clients. Treating all three as a leak can lead to restarting rather than fixing the policy that retains state.

For a hypothetical 100,000 connected clients, 16 KB of fixed per-connection state consumes 1.6 GB before payload queues. If 5% are slow and each retains 1 MB of output, they add 5 GB. Bounding per-client queues at 100 KB reduces that potential retained component to 500 MB, while requiring a disconnect or resynchronization policy when the limit is reached.

The numbers show why limits must connect to semantics. Dropping arbitrary bytes is not a valid response to memory pressure on an ordered protocol. The resource policy chooses a complete transition: reject new work, coalesce permitted state, disconnect and replay, or rebuild a snapshot.

Admission can use estimated and actual cost

A request-rate quota treats a tiny lookup and a large export equally. Cost-aware admission estimates work from query shape, expected rows, result limits, and known tenant behavior. It can reserve a budget before execution and reconcile actual consumption afterward.

estimate = classify(request, tenant_history)
reserve tenant and global budget for estimate
execute with hard memory, time, and output bounds
charge measured resource use
release unused reservation
record estimation error for future decisions

The model must handle underestimates without allowing unlimited growth. A query can spill, pause at safe points, request additional budget, or fail under a documented limit. Holding transaction locks while indefinitely waiting for more budget can create a different outage, so resource acquisition ordering matters.

Overestimation also has costs. If every request reserves its worst imaginable memory, utilization collapses. A practical policy can combine conservative hard limits with statistical reservations and controlled overcommit. The acceptable risk depends on whether exceeding memory causes a graceful query error or a process-wide failure.

Fairness requires an accounting interval. A tenant consuming a large burst should not be punished forever after it stops, but a short interval can permit repeated bursts that overwhelm neighbors. Token buckets, rolling usage, and concurrency limits solve different aspects and can be combined when justified.

Maintenance is a producer-consumer system

Foreground writes create maintenance obligations. A memtable flush produces files that compaction later merges. Updates create dead versions that cleanup later removes. Checkpoint policies require dirty pages to be written. Sustainable ingestion means the background system retires obligations at least as fast as they accumulate over the long term.

Suppose compaction debt grows by 50 MB/s during a busy period and shrinks by 25 MB/s during quiet periods. Ten busy minutes create 30 GB of debt. Clearing it needs twenty quiet minutes, assuming those rates remain stable. If the product’s normal daily schedule provides only five quiet minutes, debt grows across days even though each traffic peak ends.

RocksDB’s write-stall controls illustrate the need to limit ingestion when maintenance falls behind. Removing the limit can make the foreground chart look better temporarily while consuming remaining memory or disk space. The control protects a long-term invariant of the representation. RocksDB write stalls

Maintenance scheduling should preserve both deadlines and sufficient free space. Compaction may temporarily need output space before old files can be removed. A nearly full device can therefore be unable to perform the very cleanup that would reclaim space. Free-space reserves are operational capacity, not wasted storage.

Tenant placement and elephant tenants

Equal tenant counts do not imply equal load. One tenant can have millions of rows, broad reports, or a popular realtime channel. A placement policy based only on bytes can put several CPU-heavy tenants together while leaving other nodes underused.

Track a tenant’s resource vector and movement cost. A large cold tenant may consume storage but little CPU. A small hot tenant may dominate execution. Moving the cold tenant might free bytes without solving the hot node’s latency problem.

Splitting a large tenant can help only when its operations can divide. If every request updates one tenant-global counter, splitting records around it leaves that counter as the authority bottleneck. Replacing it with independently allocated rights or an approximate summary changes the protocol and must preserve the product contract.

Placement itself can create interference. Two tenants with complementary peak hours may share well, while synchronized daily imports can collide. Historical averages can mislead if growth changes those schedules. The placement system needs margins and safe movement rather than pretending it can predict every burst.

Autoscaling has a time-to-usefulness curve

Provisioning a node is not the same as adding useful capacity. The node must receive data or metadata, join safely, warm caches, and take ownership. During this process it can consume resources from existing nodes.

Imagine a new replica needs 600 GB and the cluster can spare 300 MB/s for copying. The byte-transfer floor is 2,000 seconds, roughly 33.3 minutes. At a continuing write rate of 20 MB/s, the catch-up design must also handle writes arriving during the copy. A naive process that copies a stale image and only then begins recording changes can lose the transition.

Autoscaling should begin with enough lead time and spare capacity to complete the transition. Triggering only at 99% utilization may be too late: the cluster has no safe bandwidth to move data. Scaling policy therefore includes forecast, minimum headroom, and admission during movement.

Scale-down has its own risks. Removing a node can increase network distance, reduce failure tolerance, evict warm caches, and trigger repartitioning. Short idle periods may not justify the next warm-up cost. Hysteresis avoids repeatedly paying transition work.

A comparative operational scenario

A SaaS platform sees a doubling of p99 transaction latency every weekday at noon. Application CPU is at 40%, so the initial suggestion is more application replicas. Trace data shows durable commit waits increasing alongside device queue depth. A scheduled tenant export overlaps compaction and backup.

More application replicas would raise potential database concurrency without increasing the constrained device bandwidth. Better interventions include moving the export to a compatible replica, limiting its read rate, rescheduling backup, or provisioning storage whose mixed-workload behavior meets the objective. Each should be evaluated against freshness and recovery requirements.

The case also shows why separate execution pools are incomplete isolation. If reports and transactions share the same devices, report I/O can still delay commit. A resource boundary should identify all shared paths. PostgreSQL’s resource settings expose several budgets, but configuration values must be interpreted within the engine’s architecture rather than copied as universal tuning advice. PostgreSQL resource consumption

Capacity as an explicit promise

A useful capacity statement names a workload envelope: operation mix, skew, data size, result sizes, durability, tenant count, and failure state. It states the latency objective and the maximum admitted rate under those conditions. It also names the transitions that need reduced traffic.

Such a statement is narrower than “supports a million users,” but it is far more informative. A million mostly idle connections and a million simultaneous contended writes are different resource demands. The engineer’s task is to translate the product’s population into actual work and preserve useful service when that work varies.

A resource-budget table makes assumptions reviewable

The earlier illustrative mix can be expressed as a table. Each row is a rate multiplied by a resource coefficient; columns remain separate because CPU, bytes, and operations are not interchangeable.

Table: A constructed resource envelope. It is a model to calibrate against measurement, not a product capacity claim.

The table immediately reveals omitted assumptions. How much of the report scan is cached? Are read misses random or sequential? Does write CPU include index maintenance? Does replication payload count one or every network hop? Asking these questions improves the model before an expensive scaling decision.

A failure-state version of the table should add recovery traffic and reduce available resources. A migration-state version adds backfill and validation. A reconnect-state version adds snapshots and authentication. These are separate workload envelopes rather than arbitrary safety multipliers applied to a healthy average.

Capacity planning becomes useful when it predicts which limit will be reached first and what intervention changes that limit. Its purpose is not numerical precision unsupported by evidence. It is to make the causal relationship between demand, resource consumption, and useful service explicit enough to test.

Case connection and design conclusion

PostgreSQL’s resource settings and RocksDB’s maintenance controls demonstrate different resource-management surfaces. Their configuration values are not interchangeable, but both show that foreground query speed depends on budgets shared with other engine work.

A capacity model should connect workload rates to resource vectors, include skew and maintenance, and reserve enough transition capacity for the promised failures. Lesson 36 explores architectures that separate compute and storage so some dimensions can scale independently while communication and cache behavior create new coupling.

Sources and further reading

• PostgreSQL 18 resource consumption
• RocksDB write stalls

Course index · Lesson plan