Geographic architecture — extended reference

This reference preserves the earlier detailed chapter and its citations. Start with the guided explanation, then use these sections for additional mechanisms and qualifications. References to the original lesson sequence are historical.

Part 7: Building and operating applications on these engines

Prerequisites: Lesson 37 and the preceding foundations.

Learning outcomes

• Allocate a geographic latency budget across ownership, execution, replication, and client delivery.
• Compare regional ownership, global reads, and global writes under the required consistency contract.
• Explain how failover placement changes both latency and failure independence.

A global user base does not require every decision to be global

A marketplace serves buyers in three regions. Product descriptions can be cached widely. A tenant’s private records can usually stay near its home region. A unique global item requires an authoritative ownership decision. Placing all three workloads behind the same global write path can spend latency where the product does not need it.

Geographic architecture assigns data, computation, and authority across distance. It must connect consistency requirements to communication paths and failure domains. A map of regions is not enough; the design needs to show where each operation’s required evidence resides.

Regional ownership

A tenant or account can have a home region that owns its writes. Requests near that region follow a short path; distant requests pay transport delay. Regional ownership can simplify ordering and reduce cross-region coordination for common operations.

Migration changes ownership and needs fencing, routing updates, and state transfer. An old region must not continue accepting writes after authority moves. Clients with cached routes need a safe redirect or retry path that preserves operation identity.

A tenant-local invariant can scale across many independent home regions. A cross-tenant transfer can require another protocol. The application should identify how often the cross-owner path occurs and whether it is acceptable for it to have a different latency budget.

Replica placement and write paths

Replication across failure domains can survive larger failures, but synchronous acknowledgment across distance adds waiting. Waiting for a nearby quorum differs from requiring a remote region. Placement rules determine which failure survives and which paths dominate latency.

An illustrative operation with two sequential 80-millisecond WAN round trips spends 160 milliseconds on those dependencies before processing. Reducing local execution by 5 milliseconds helps but cannot remove the 160-millisecond path. Overlapping independent work can shorten elapsed time only when the protocol permits it.

A topology diagram should distinguish client-to-leader distance, leader-to-replica distance, and transaction-participant distance. These are different paths and may have different congestion and failure behavior. A benchmark’s region name alone does not specify them.

Global reads and freshness

Read-mostly data can be replicated or cached near users. The freshness contract determines whether a local answer is sufficient. A static catalog image, a recent analytics projection, and a current account balance may require different guarantees.

Read-your-writes can use a session watermark and wait for a local replica, or route to a sufficiently current authority. Bounded-staleness reads need a way to verify the bound under partitions and clock assumptions. A replica being nearby does not establish that its state is recent enough.

A globally consistent snapshot across several regions needs a shared interpretation of transaction order or timestamps. The system may wait for safe progress or choose an older timestamp known to be available. That changes freshness and latency together.

Data residency and access boundaries

Placement can be constrained by legal, contractual, or organizational requirements. This course does not give legal advice; it treats such requirements as explicit inputs to architecture. Copies in backups, logs, caches, support tools, and disaster-recovery locations must be included in the data-flow model.

A primary region label does not prove that no other copy exists. Encryption also does not automatically satisfy a requirement about where data or keys reside. Operational metadata can itself contain identifiers or sensitive values. The architecture should document relevant classes of data and their allowed movement.

These constraints can limit failover choices. A service may prefer temporary unavailability to moving protected data to an unauthorized location. That is a legitimate contract decision that should be designed and communicated before an incident.

Quotas and independent rights

A global stock count can sometimes be divided into regional selling rights. If region A owns twenty units and region B owns thirty, each can accept local sales within its allocation without contacting a global counter for every purchase.

The sum of rights must remain bounded by actual stock, and moving rights requires coordination. One region may temporarily reject purchases while another has unused rights. The product trades perfect global utilization for local decisions and partition tolerance within allocations.

This approach works only when rights can be represented and transferred safely. A unique indivisible item cannot be simultaneously allocated to several regions. A financial or inventory model with reversals and returns needs additional rules. The optimization changes where authority lives, not the need for authority.

Failover and degraded geography

Failover can preserve service while moving users farther from the new authority. Latency objectives may therefore differ between healthy and regional-failure modes. A service should state whether the failure objective concerns correctness, availability, latency, or all three.

Promoting a remote replica requires a safe history and fencing of the old region. Catch-up and cache warming can consume the same network needed for user traffic. A regional outage can also cause client retries and routing churn, making load larger than its ordinary regional share.

Recovery placement should preserve enough headroom for this transition. If every surviving region is already near capacity, a correct failover can still create severe overload. Geographic redundancy and geographic spare capacity are separate design dimensions.

Historical and current case connections

Spanner’s original paper describes a globally distributed database with explicit mechanisms for time and transactions. CockroachDB’s multi-region documentation describes placement and locality controls in its product architecture. Their specific mechanisms should be read in their version and historical context.

The broader inference is that global consistency is most affordable when common operations align with ownership and replica placement. A system can correctly support expensive global transactions while encouraging local schemas for ordinary work. Capability and preferred workload shape are different questions.

A geography diagram needs three kinds of arrows

A common architecture picture puts one database icon in each region and labels the system global. It omits the dependencies determining latency. Draw client-to-entrypoint, entrypoint-to-authority, and authority-to-required-replicas separately. A transaction involving multiple owners needs a fourth set of participant arrows.

client in region A
   -> gateway A
       -> transaction authority B
           -> replica quorum B/C
           -> participant authority D, if required

Figure: Local connection acceptance does not imply local authoritative execution. Each required cross-region arrow belongs in the latency budget.

A nearby gateway can reduce connection setup or terminate encryption locally while forwarding the actual transaction elsewhere. A nearby replica can serve historical reads while current reads still communicate with authority. These are useful capabilities, but their endpoint names should not be mistaken for a complete placement contract.

The budget also distinguishes sequential from parallel arrows. A coordinator contacting two remote participants concurrently waits for the slower path. A later decision phase may add another round trip. Counting every arrow as sequential overstates latency, while counting all communication as parallel ignores protocol dependencies.

Distance imposes a physical floor

Light in fiber travels at roughly two hundred thousand kilometers per second before routing and equipment overhead. A 10,000 km one-way physical path therefore contributes about 50 ms, or about 100 ms for a round trip under this simplified model. Actual routes can be longer than map distance and include queueing, serialization, and processing.

This is an order-of-magnitude physical illustration, not a promise about any pair of regions. It explains why eliminating 2 ms of local execution cannot make a protocol with a mandatory intercontinental round trip complete in 5 ms. The architecture must remove or relocate the dependency, overlap permitted work, or change the contract.

Replication geography and client geography create different costs. A client near the leader can still wait for a distant required replica. A client far from a leader can pay distance even when all replicas are close to that leader. The optimal placement depends on both request origins and failure requirements.

A latency objective should therefore specify locations and operation classes. “Under 20 ms globally” is not a complete requirement for exact coordinated writes. It may be achievable for cached reads while impossible for a particular global transaction.

Regional ownership as a directory problem

Assigning each tenant a home region keeps common operations near one authority. The service needs a directory mapping tenant identity to current ownership and an epoch identifying that assignment. Clients and gateways can cache the mapping, but stale entries must be safely rejected or redirected.

Moving a tenant from A to B requires more than copying rows. New writes during copying must be captured. A final transition fences A, establishes B’s authoritative history, and publishes the new route. A request with an old route should retain its operation identity through redirection so a lost reply does not become a duplicate mutation.

copy consistent state A -> B
stream changes after copy boundary
fence writes under epoch e through authoritative ownership protocol
catch B up through the final accepted prefix
enable B under durable ownership epoch e+1
publish B as owner
resolve in-flight requests under stable operation identities

This is a conceptual outline. A real protocol must define who durably decides the transition and how crashes at every step resolve. The key invariant is that both regions cannot independently accept conflicting writes under different beliefs about ownership.

Tenant mobility can be useful when usage patterns change, but frequent movement consumes bandwidth and cache warmth. A placement controller needs hysteresis and should avoid moving authority in response to short transient traffic shifts.

Bounded staleness needs evidence

A local replica advertised as “at most five seconds old” needs a way to establish that bound. Knowing that its last applied transaction has a timestamp near now may be insufficient if no transactions occurred recently or if clock uncertainty is uncontrolled. The replication protocol can provide a safe progress position or heartbeat meaning.

During a partition, a replica may continue serving old state. It cannot preserve a strict staleness bound indefinitely without either new evidence or refusing the bounded-stale request. Continuing to answer while silently exceeding the bound changes the contract.

Read-your-writes is narrower. A client can carry the position of its own completed write and require a server to serve at least that position. A local replica may wait until it catches up or route the read to another suitable server. This can offer useful session behavior without promising every client’s read is globally freshest.

Historical snapshots are another interface. A query can intentionally choose a stable timestamp available across participating replicas. The result is consistent at that time even when newer state exists. “Consistent” does not automatically mean “latest.”

Allocate rights when the invariant permits it

Consider stock of 100 interchangeable units. A central process can allocate 40 selling rights to region A, 35 to B, and 25 to C. Each region decrements its local available rights for accepted purchases. As long as rights are not duplicated and local spending is protected, total accepted sales cannot exceed 100.

During a partition, A can continue selling its remaining rights without contacting B or C. If A sells out while C has unused rights, A may reject demand despite global remaining stock. The design trades utilization flexibility for local availability and latency.

Transferring rights must be atomic with respect to their ownership. If A sends ten rights to B and crashes, the protocol must not let both count them or neither recover them indefinitely without a defined resolution. An escrow-style representation can make transfers explicit, but it still needs a durable authority model.

Returns also require rules. A returned unit can restore rights in the region that owns the return, or enter a reallocation pool. Reusing a returned unit before the refund and cancellation workflow is resolved can violate other business assumptions. Local rights simplify one invariant, not every surrounding workflow.

For a unique sword, allocating one right to each region would duplicate authority. Only one region can hold its selling right at a time. The optimization is constrained by divisibility of the business resource.

Global uniqueness is an often-hidden requirement

A username, coupon code, or external identifier may need uniqueness across all regions. If every region independently accepts the same value, later reconciliation cannot make both original success responses true under a strict uniqueness contract.

Possible designs include one authoritative namespace service, partitioning by normalized identifier, allocating disjoint identifier ranges, or allowing provisional names that may later change. Each moves coordination or changes the user experience. A globally unique random identifier can avoid most coordination for generated identities, but user-chosen names require conflict handling.

Normalization is part of the invariant. Case folding, Unicode handling, and whitespace rules must agree everywhere. Two regions that normalize differently can violate uniqueness even with reliable replication. This is a reminder that distributed correctness includes application semantics and encoding, not only consensus.

A global namespace can also become a hotspot during a launch. Partitioning by identifier distributes independent names, but one popular requested name still needs one decision. Product design can return a definitive conflict rather than repeatedly retrying an impossible claim.

Failover placement changes the user journey

Suppose tenant authority normally lives in region A, with recoverable replicas in B and C. Losing A can preserve data while moving execution farther from its users. If ordinary client-to-authority RTT is 8 ms and failover RTT is 90 ms, every dependent authoritative exchange becomes more expensive.

A healthy-state 20 ms p99 objective may not be realistic during this failure. The service can define a separate degraded objective, maintain a nearer eligible replica, or reduce dependent exchanges. The choice should be deliberate and tested.

Failover can also overload B. It receives A’s users, reconstructs caches, applies remaining log, and rebuilds redundancy. If B was already near capacity, geographic resilience may preserve bytes but fail to preserve useful service. Capacity planning must model the location and workload distribution after failure.

A recovery procedure must fence A if it later returns. Merely changing client routing is insufficient if old processes can still accept direct requests or perform external effects. Authority belongs to a protocol, not to whichever hostname a client happened to resolve.

Data movement is broader than table placement

A tenant’s primary rows can remain in one region while backups, logs, telemetry, support exports, and caches copy identifiers elsewhere. A placement requirement must define data classes and all representations. This is an architectural inventory, not an interpretation of any particular law.

Encrypted data still needs placement and key-management decisions. If a regional recovery copy survives but its only key service was lost in the failed region, the copy may be unusable. If support logs contain plaintext payloads, encrypting table files does not protect that path.

Operational metadata can be sensitive. Query text, error messages, trace attributes, and object names can reveal user information. A geographic design should specify permitted movement and retention for these classes, rather than assuming “metadata” is automatically harmless.

The failover policy must respect the same constraints. A system may intentionally refuse writes when no authorized location can safely become authority. That is a real availability limit that should be stated before an incident.

A worked decision for a global marketplace

A hypothetical marketplace has three data classes. Catalog text changes infrequently and tolerates one-minute freshness. Individual seller inventories require exact ownership. Aggregate sales dashboards tolerate ten-minute freshness. Placing all three on one globally synchronous path would pay unnecessary latency for catalog and reporting.

A suitable design can serve catalog replicas near readers, keep inventory authority aligned with sellers or listing partitions, and feed an analytical representation asynchronously. Cross-owner settlement uses the required exact protocol. The user interface distinguishes inventory confirmation from approximate displayed availability.

The cost is multiple freshness contracts and maintained pipelines. The catalog cache needs invalidation or expiry. Analytics needs recoverable ingestion positions. Inventory authority needs retries, routing, and failure-safe ownership. This is still simpler than forcing every read into the strongest path if the application can express these distinctions clearly.

Spanner’s historical paper and CockroachDB’s current multi-region guidance illustrate concrete ways distributed databases organize locality and consistency. The scenario above is original architectural analysis rather than a claim that either product automatically chooses this design. Spanner paper, CockroachDB multi-region overview

Geography as a correctness and UX decision

Users experience the placement contract through response delay, stale displays, pending operations, and degraded behavior during outages. An honest pending state can be preferable to an immediate false success. A locally fast historical view can be useful if its freshness is visible and it is not used for an exclusive decision.

The engineering task is to decide which facts require immediate agreement, which can be owned regionally, and which are derived. Then place authority, replicas, computation, and caches to match those contracts. Geography becomes manageable when the dependency graph is explicit, even though distance and failures remain unavoidable.

A geographic latency table for one operation

The following hypothetical comparison holds local execution at 4 ms and changes only the required communication path. It illustrates dependencies rather than measured regional performance.

Table: Local execution improvements affect the 4 ms term; placement and protocol change the communication terms. Real paths can overlap selected stages.

A valid optimization proposal identifies the row it changes and the guarantee it preserves. Moving the leader nearer the client may reduce the first RTT while moving it farther from required replicas. Dropping the distant copy from the acknowledgment policy may improve delay while weakening the regional-loss contract. Serving an older local snapshot changes the read interface rather than accelerating the same fresh read.

The same table can include a failover row. If authority moves across an ocean after a regional failure, the healthy-state target no longer describes the operation. Publishing a degraded target helps application teams choose deadlines and UI states that remain useful during recovery.

A product can also separate the user’s immediate interaction from global completion. A request may receive a durable local pending status while a globally coordinated transition continues. That can improve perceived responsiveness when the pending state is accurate and recoverable. It does not justify calling a provisional decision final.

Geographic design therefore belongs in curriculum discussions of UX as well as distributed systems. Users experience whether a state is pending, stale, confirmed, or unavailable. Clear interfaces can make unavoidable distance understandable, while vague success semantics convert physical delay into correctness problems.

The architecture should end with a small set of named contracts rather than one unqualified “global” label. Each contract states authority, freshness, durable copies, failover behavior, and expected path. That vocabulary makes later performance and incident discussions much more precise.

The operation’s contract should remain attached to every published latency number. Location, freshness, required copies, and failure mode are part of the result, not optional benchmark footnotes.

Design conclusion

A geographic design maps operation classes to authority, replica placement, freshness, and failure behavior. It counts sequential WAN dependencies and explains which product contracts allow local decisions. Lesson 39 then follows the system through loss, restoration, and the operational steps required to regain useful service.

Sources and further reading

• Spanner original paper, Google, 2012
• CockroachDB multi-region overview

Course index · Lesson plan