Realtime delivery — extended reference

This reference preserves the earlier detailed chapter and its citations. Start with the guided explanation, then use these sections for additional mechanisms and qualifications. References to the original lesson sequence are historical.

Part 6: Scaling execution and state delivery

Prerequisites: Lesson 32 and the preceding foundations.

Learning outcomes

Establish a race-free relationship between an initial snapshot and subsequent updates.
Explain event order, replay, filtering, fan-out, and authorization changes across reconnects.
Bound slow-consumer memory and choose resynchronization behavior when history is no longer retained.

The missing update between snapshot and subscription

A client fetches the current inventory at time A. An item changes at time B. The client starts its change subscription at time C. If the snapshot did not include B and the subscription begins after B, the client never learns the change. Polling less often or using a persistent connection does not resolve the gap by itself.

Realtime delivery maintains client-visible state over a changing server state. It needs a coherent initial snapshot, an ordered or otherwise interpretable stream, reconnect behavior, retention, permissions, and backpressure. A message broker can supply useful primitives while the application still owns the state protocol.

Events and state are different interfaces

An event announces an occurrence: an item moved, a message was sent, or a notification should play. State describes the current result: the item’s owner, the current message collection, or the latest position. A stream of events can reconstruct state only if its semantics, completeness, and ordering are sufficient.

A current-state subscription may coalesce intermediate updates while preserving the latest visible value. An audit stream may need every event. Dropping obsolete movement updates can be acceptable; dropping one payment event from an accounting history is not. The delivery policy follows the purpose.

A client cache should identify records stably so updates and deletions apply to the correct objects. A result row without a stable identity may need insert/delete semantics rather than an in-place update. Query changes and overlapping subscriptions can also affect which rows remain in the union of locally retained data.

A watermark aligns snapshot and changes

A safe conceptual protocol obtains a snapshot corresponding to position S and then delivers every required change after S, or establishes the stream first and reconciles an aligned snapshot. The exact implementation can use transaction logs, sequence numbers, or another consistent boundary.

server snapshot: state through position S
client installs snapshot and records S
server replays required changes S+1 onward
client advances applied position after applying each change

The schematic assumes the stream positions describe the same authoritative history as the snapshot. It also assumes enough history is retained. A reconnecting client too far behind may need a new snapshot rather than arbitrary partial replay.

Changes can be grouped by transaction to preserve a coherent visible transition where promised. Showing the debit before the matching credit can temporarily violate a display invariant even when server state is atomic. The client SDK and callback semantics should define the boundary at which application code observes the batch.

Ordering and duplicate handling

Per-key order, per-channel order, transaction order, and global order are different guarantees. A stream partitioned by key can preserve each item’s sequence while events for different items interleave differently. A cross-item transaction needs extra metadata if clients must observe it atomically.

Retries and reconnects can replay messages. Applying an update with a stable version can be idempotent; incrementing a counter for every delivery is not. A client should reject obsolete versions or follow a protocol that establishes exact replay boundaries. Sequence gaps need detection and recovery rather than silent continuation.

A delivery acknowledgment should identify what it means: received by a socket, decoded, applied to local state, or durably stored on the client. A server using acknowledgments for retention must choose the appropriate boundary. A client that acknowledges before applying can lose state on its own crash.

Fan-out arithmetic

At an illustrative 20,000 changes per second, 100 recipients per change, and 80 payload bytes per recipient, payload delivery is 20000 × 100 × 80 = 160000000 bytes/s, or 160 MB/s. Serialization, framing, encryption, acknowledgments, and copies add further work.

One shared query computation can reduce repeated evaluation but does not remove per-recipient network bytes. Compression or shared serialized buffers can reduce selected costs while requiring lifetime management. A system can saturate network capacity even when transaction execution remains fast.

Interest management limits recipients to those who need an update. Geographic regions, rooms, tenants, topics, or authorized query predicates provide selection boundaries. Smaller groups reduce fan-out but increase membership changes and metadata. A player moving across many tiny regions can create subscription churn.

Slow consumers and bounded state

A client with poor connectivity may receive changes slower than they are produced. If the server retains every undelivered update indefinitely, one client can consume unbounded memory or log retention. The protocol needs limits and a defined fallback.

For a current-state feed, coalescing newer values can bound some per-key backlog. For an ordered event history, the system may disconnect the client and require replay from retained storage. A client beyond retention needs resynchronization. These are product-visible behaviors and should be designed deliberately.

Backpressure should propagate to the right scope. One slow client should not block unrelated clients or hold a transaction lock while its network drains. Shared queues need fairness and separate accounting. A large subscriber population can make connection metadata itself a significant memory cost.

Permissions during long-lived subscriptions

Authorization is not only a connection-time check. A user’s membership can change while the connection remains open. The system must decide how existing subscriptions are invalidated and whether previously delivered data remains on the device.

A client-selected filter is not a security policy. An untrusted client can request a broader query unless the server enforces visibility. Query evaluation, cache sharing, and replay must all respect the same access boundary. Shared serialized results must not contain fields that one recipient is not allowed to see.

Retention also has privacy implications. A user removed from a room should not regain old private data merely by replaying a channel with a stale token. The protocol must bind access checks to current or explicitly versioned policy as required by the application.

Case comparison

PostgreSQL logical decoding exposes committed change information with its own interface and operational responsibilities. InsForge documents channel delivery through database-trigger publication and broadcasts. SpacetimeDB documents subscribed row state and local caches. These are distinct abstractions around communicating change.

The architectural analysis is that an integrated subscription model can remove application code for maintaining selected row state, while channel events offer flexible signaling. Neither abstraction eliminates the need to reason about retention, permission changes, recipient count, and client processing delay. Their suitability depends on whether the product needs current state, event history, or both.

A complete snapshot-and-stream trace

Consider a hypothetical inventory service with committed positions 410, 411, and 412. At 410 the player owns a shield. Transaction 411 trades the shield away and awards a sword. Transaction 412 changes the sword’s durability. The reconnecting client has no usable local state and requests a fresh subscription.

The server first establishes a consistent snapshot through 410 and guarantees retention or buffering of relevant changes after that boundary. While it serializes the snapshot, transactions 411 and 412 commit. It sends the snapshot with its boundary, then the change batches in the authoritative order. The client installs the snapshot before applying 411 and 412. The final state matches the subscribed projection after 412.

Now consider the unsafe variation. The server serializes the inventory, starts watching only after serialization finishes, and labels neither step with a shared position. Transaction 411 occurs between the two. The client keeps the shield and never receives the sword, while later durability updates may reference an object it does not know. A reliable connection faithfully delivers an incomplete protocol.

The safe trace requires an actual consistency relationship. A wall-clock timestamp attached after serialization is not automatically a snapshot watermark. If different rows were read at different states, the snapshot may never have existed. The engine must use its snapshot mechanism or another protocol establishing an equivalent boundary.

A subscribe-first design can also be correct. It begins capturing changes, obtains a snapshot aligned with a known point, and discards or reconciles changes already represented in that snapshot. The choice affects buffering, concurrency, and complexity, but both designs must prove completeness and avoid double application.

Projection changes include removals

A filtered subscription is not merely a feed of writes whose new values match the filter. Suppose a client watches items whose owner is player 7. An item moving from player 7 to player 9 must disappear from the result. The new row no longer matches, but the transition remains relevant.

Conceptually, evaluate membership before and after:

old_matches = predicate(old_row, authorized_context)
new_matches = predicate(new_row, authorized_context)
if not old_matches and new_matches: insert result
if old_matches and new_matches: update result
if old_matches and not new_matches: remove result

This is a teaching model for a row-local predicate. Joins and aggregates need additional maintained dependencies. A membership table changing can alter results for many underlying rows even when those rows themselves do not change. Correct incremental evaluation follows dependencies rather than only the directly written table.

Deletion metadata also deserves care. A removal must identify which local result to remove without disclosing private new values. Sending an entire post-transfer item record to the previous owner merely to communicate its disappearance can leak the new owner’s private attributes. The state protocol and authorization model must agree on the information in each transition.

Overlapping subscriptions add reference accounting. If one subscription covers region A and another covers nearby friends, a row may belong to both. Removing the row from the first result must not remove it from the client’s union while the second still includes it. The SDK can manage this, but the conceptual requirement explains why local cache behavior may differ from a simple array of last messages.

Join maintenance can amplify one small write

Suppose an authorized query joins room membership with public avatars. One membership change can affect hundreds of visible avatars for one subscriber. Conversely, changing one public avatar can affect every room query including that avatar. The directly written row count is therefore not the same as the result-change count.

An illustrative game has 2,000 users in one public room. One avatar update is delivered to 1,999 other users. At 120 bytes per payload, that produces about 239,880 bytes before protocol overhead. At ten such avatar changes per user per second, unrestricted all-to-all delivery becomes approximately 2,000 × 10 × 1,999 × 120 = 4,797,600,000 B/s, about 4.8 GB/s. The model intentionally assumes full mutual visibility; interest management changes the multiplier.

A shared query structure can avoid repeating the same membership calculation, but it cannot remove those recipient bytes. A smaller room, coarser update frequency, compressed representation, or different visibility contract reduces a different part of the cost. Stating exactly which factor changes prevents vague claims about “optimizing realtime.”

The same amplification appears outside games. A widely shared organization setting can invalidate many personalized dashboards. A popular document permission change can alter many subscriptions. Dependency-aware systems make these changes easier to express, while capacity planning still needs the size of the affected dependency graph.

Client application is another execution stage

A server can deliver updates faster than a browser or game client can apply and render them. Decoding, object allocation, collection updates, framework notifications, and layout all consume client resources. A network acknowledgment does not prove the display has caught up.

Assume a client receives 5,000 row updates/s and spends an average 80 microseconds applying each update plus reactive notifications. That is 0.4 CPU-seconds/s before rendering. If updates arrive in a batch of 1,000, applying the batch can occupy 80 ms of one execution thread. Average utilization may look acceptable while the application visibly freezes.

Batch-aware callbacks can reduce repeated recomputation. Instead of rebuilding a sorted view after every row, the client can update its storage and notify the presentation layer once per coherent batch. The API must preserve the necessary observation boundary: application code should not see an inventory in an impossible half-applied transaction if atomic client transitions are promised.

Rendering can use interpolation or sampled display updates while retaining exact logical state underneath. That is a presentation policy, not permission to drop arbitrary durable events. A client should distinguish its current applied server position from its latest rendered frame so diagnosis can locate the lag.

Backlog policies follow semantics

There are at least three useful queue policies. A latest-state queue can replace an obsolete pending value for the same entity. A transaction-preserving queue retains ordered batches. An audit queue retains every required event until durably acknowledged or moves delivery into a replayable log.

Coalescing works only if intermediate values have no independent meaning. Replacing position 10 with position 20 may be acceptable for a display. Replacing “charge 10” with “charge 20” is not equivalent to the two events. Even state coalescing can be unsafe when a client relies on transitions, such as entering and leaving a trigger region.

A bounded queue needs a defined overflow outcome. Disconnect-and-resnapshot can be correct for a current-state view, because the new snapshot replaces missing intermediate states. For an audit stream, reconnect must replay the missing events from retained history. If history is gone, the interface must report a gap rather than silently continue.

Retention costs can be calculated. At 2 MB/s of retained stream data, keeping one hour requires 7.2 GB before indexes and replication. If one slow subscriber holds the minimum retained position for all subscribers, its delay can impose that storage cost on the service. Segment-level retention, separate subscription classes, or disconnect policies can bound coupling.

Authorization transitions need an explicit event

A subscription established under yesterday’s permissions may remain connected after membership is revoked. The server must re-evaluate or invalidate access according to the contract. It cannot rely on an honest client to unsubscribe itself.

Revocation also has an unavoidable limit: previously delivered plaintext cannot be made unknown to the recipient. Removing a row from a local cache can support UI behavior, but a modified client may have retained a copy. Sensitive data should not be delivered before the user is authorized merely because the interface hides it.

A generic secure transition increments a policy version, updates membership transactionally, and causes affected query results to be recomputed or terminated. Replay requests are checked against current authorized scope or an explicitly permitted historical scope. A stale connection token must not silently restore access to an old private channel.

Shared result caching needs equivalent discipline. Two users with identical textual queries may have different authorized results. Sharing based only on query text can leak data. The cache key or shared-evaluation mechanism must include the relevant authorization context or operate on a demonstrably public result.

Choosing an abstraction through a product scenario

A hypothetical chat application needs message history, typing indicators, unread counts, and room membership. Message history is durable state with stable identifiers. Typing is ephemeral and naturally expires. Unread counts are derived state with a defined boundary. Membership is security-sensitive authoritative state.

A channel broadcast is an appropriate tool for typing because missing an old indicator is harmless. A subscribed message query can maintain history and pagination state. A durable event log may serve moderation or integration consumers. Treating all four needs as one undifferentiated WebSocket feed creates unnecessary complexity.

PostgreSQL logical decoding exposes a committed-change interface; InsForge’s channel publication provides application-oriented event delivery; SpacetimeDB subscriptions supply row-state synchronization. The teaching comparison concerns ownership of the state protocol, rather than an assertion that one abstraction has every feature the others lack. Logical decoding, InsForge realtime, SpacetimeDB subscriptions

The product design should explicitly choose replay requirements, initial-state semantics, ordering scope, authorization, and overflow behavior for each interface. That decomposition often saves more engineering effort than trying to optimize a single universal feed.

A practical latency budget for delivery

Assume an illustrative action spends 2 ms waiting for execution, 1 ms executing, 3 ms at its durability boundary, 2 ms in subscription evaluation and batching, 20 ms crossing the network, and 4 ms applying on the client. The visible state-update delay is about 32 ms if the stages are sequential. Reducing execution to 0.2 ms saves only 0.8 ms of that total.

If the delivery batch waits another 30 ms during low traffic, the user sees roughly 62 ms. A benchmark measuring only action execution would miss the dominant new delay. Bounded batching should therefore be selected against the end-to-end product budget, not only server throughput.

A useful trace carries one operation identity through commit and delivery positions where the product permits it. It records when the client applies the relevant batch. This makes it possible to distinguish a fast server with slow fan-out from a slow action with prompt delivery. Both create stale displays, but their remedies differ.

Realtime systems succeed when their state protocol is complete and their resource costs are bounded. A persistent socket is only the transport. The actual engineering work is aligning histories, selecting authorized results, controlling amplification, and ensuring clients can keep up.

Reconnect storms are a distinct workload

Suppose a deployment disconnects 50,000 clients within one minute. Each needs an initial snapshot of 200 KB. Reconstructing those snapshots creates about 10 GB of payload before protocol overhead. Delivering all of it in ten seconds demands 1 GB/s, even if ordinary incremental traffic is much smaller.

The server also evaluates initial queries, authenticates connections, allocates cache metadata, and potentially performs overlapping scans. A healthy steady-state subscription benchmark does not cover this event. If every client retries at the same fixed interval, the burst repeats and may prevent recovery.

A reconnect policy can use randomized backoff, admission limits, reusable public snapshot material, and resumable positions where retained history permits it. Reuse must preserve authorization: a public region snapshot can be shared more broadly than a private inventory snapshot. Compression reduces bytes but consumes CPU and may add latency for small results.

Resumption is often cheaper than full reconstruction, but it is not always preferable. A client missing a million changes to a state containing only a thousand rows may be better served by a fresh snapshot. The server can choose a crossover based on retained history, expected bytes, evaluation cost, and client capability. The choice should retain the same final-state guarantee.

This recovery workload belongs in capacity planning alongside normal fan-out. A system can be comfortably provisioned for small deltas yet unable to reconnect its population after one regional interruption. The correct service objective measures time until clients regain coherent useful state, rather than only time until the first WebSocket opens.

The general pattern recurs throughout distributed systems: synchronization work often grows when the system is already degraded. Safe admission and retained recovery state are therefore part of the realtime architecture, not optional operational additions.

A useful final distinction is availability of the transport versus availability of coherent data. A connected client whose snapshot cannot complete is not fully recovered. Operational dashboards should therefore count subscriptions that have installed their initial state, their applied positions, and their lag, alongside socket counts. That makes the service’s actual synchronization progress visible.

Design conclusion

A realtime protocol should describe snapshot alignment, ordering scope, duplicate behavior, progress acknowledgments, retention, resynchronization, and slow-client policy. The fastest transaction path is only one component of making shared state useful to many clients.

Lesson 34 examines overload across these paths. An engine that can reject or slow work coherently is often more usable than one that accepts unlimited requests and lets every queue grow.

Sources and further reading

Course index · Lesson plan