Course contents
On this page
Lesson plan 25: Distributed failure models
Reason about delayed, duplicated, reordered, and lost messages without assuming a timeout proves failure.
Placement: Part 5: Correctness across machines.
Prerequisites: Lesson 24: Crash recovery and durable commit, plus the earlier dependency chain.
Learning outcomes
- Reason about delayed, duplicated, reordered, and lost messages without assuming a timeout proves failure.
- Separate physical clocks, causal order, and protocol epochs.
- Design a retry outcome that remains valid after a client or server loses its reply state.
- Separate a general mechanism, a documented implementation choice, and an illustrative numerical assumption.
Teaching sequence
- Establish the motivating situation: A paused leader is mistaken for a failed one and later resumes with obsolete authority.
- Introduce the vocabulary and the correctness obligation before an optimization. Define partial failure, delayed messages, failure detection.
- Develop the mechanism: Separate suspicion from proof, deadlines from cancellation, and epochs from wall-clock freshness.
- Follow the complete worked example: A 200 ms pause can exceed a 150 ms detector threshold without any machine crash. Explain the assumptions, intermediate units, and limiting interpretation.
- Extend the scenario to changed load, skew, or failure, explaining clocks, ordering, duplicates, ambiguity.
- Connect the mechanism to a documented implementation and an alternative with a different workload fit. Close by stating the remaining cost and the next lesson dependency.
Required chapter material
Use a complete execution trace or worked calculation, supported by an annotated static diagram or table. Include language-neutral pseudocode or labeled SQL when it clarifies the mechanism. State preconditions, ownership, visibility, and failure boundaries for algorithms. Describe operational costs as well as the normal path. All questions raised in the narrative must be resolved in the text.
Visual explanation
A partitioned-leader sequence shows the same timeout observation arising from a failed server and a delayed network, motivating fencing rather than certainty from time alone. The accompanying prose explains the arrows, state changes, or table entries so the lesson remains understandable in a plain Markdown reader.
Editorial checks
Confirm that the numerical example agrees with its units and assumptions. Check that a faster path preserves the stated contract, or explicitly identify the changed guarantee. Keep hypothetical figures separate from product measurements. Explain both an appropriate use and a limitation of the case design. Check terminology against the glossary and links against the course index.