A queue cannot absorb an endless surplus

The shop can sustainably complete 100 requests each second under a stated workload. A promotion sends 150 requests per second for ten seconds. Assume arrivals and completions are uniform and the service begins with an empty queue.

A queue can defer work, but it cannot create processing capacity. During the burst, the backlog grows by 50 requests each second.

Follow the backlog after the promotion ends

1. Ten-second burst

   1,500 requests arrive; 1,000 complete. Backlog reaches 500.

2. Arrivals fall to 80 per second

   Capacity remains 100. Only 20 completions per second are available to reduce the backlog.

3. Drain the accumulated work

   500 ÷ 20 = 25 seconds before the queue returns to empty.

A request arriving behind 500 queued requests may wait roughly five seconds for them to be processed at 100 per second, before its own service time. Variability, priorities, and concurrency can change the exact waiting time. The accumulated work remains real.

Decide what the service will accept

Offered load is what callers attempt to send. Admitted load is work the service accepts. Load shedding rejects some offered work deliberately so that accepted work can still complete usefully.

A bounded queue defines how much waiting work the engine will retain. When it is full, the caller receives a controlled overload response rather than waiting until memory is exhausted. The bound should reflect a waiting-time and resource budget, not just a convenient large number.

A deadline states when the result stops being useful to the caller. If expired work continues using resources, it can delay requests that still have a chance to succeed. Cancellation must release the resources already acquired by the executor.

Retries can extend the overload

A caller times out and sends the same operation again while the first attempt is still running. If many callers do this together, retry traffic adds to the original load. Delaying retries with randomized timing can reduce synchronization, but it does not increase the engine’s sustainable capacity. A retry also needs a rule preventing duplicate business effects.

Remove the spare capacity

After the burst, arrivals settle at exactly 100 per second. How long does the 500-request backlog take to drain under our constant-rate assumptions?

Look for a positive drain rate

It never drains: 100 arrivals consume all 100 completions each second. The backlog remains 500. Recovery requires lower admitted arrivals, greater useful completion capacity, or safely discarding work that is no longer needed.

Overload control protects progress while capacity is limited. The next lesson identifies which resource actually limits that capacity.

Google SRE’s overload chapter explains admission and overload-management concerns. The numerical queue above is illustrative rather than a measured service.

Extended reference

The extended chapter preserves the original technical coverage, additional examples, and citations. The explanation above supplies the concepts needed for this page; the reference develops the specialist cases.

Find a section from the original chapter

• Learning outcomes
• An overload can outlive its original spike
• Offered, admitted, and completed work
• Queue discipline and fairness
• Batching and pipelining
• Load shedding
• Retry budgets and randomized delay
• Tail latency mechanisms
• Recovery from overload
• Queue placement changes what failure looks like
• Deriving a queue limit from a waiting budget
• Service-time variance matters
• Deadline propagation prevents useless downstream work
• Retry amplification across layers
• Batching has an occupancy equation
• Hedging needs spare capacity and safe semantics
• A full overload-and-recovery timeline
• Fairness and recovery work
• A bounded recovery diagram
• Case connection and design conclusion
• Sources and further reading