Caches and maintained results — extended reference

This reference preserves the earlier detailed chapter and its citations. Start with the guided explanation, then use these sections for additional mechanisms and qualifications. References to the original lesson sequence are historical.

Part 6: Scaling execution and state delivery

Prerequisites: Lesson 31 and the preceding foundations.

Learning outcomes

Distinguish a cache, materialized view, incremental result, CDC stream, and CQRS representation.
Explain invalidation races, rebuildability, freshness watermarks, and transactional maintenance.
Calculate miss load and recognize stampedes, negative caching, and hot-key amplification.

A late cache fill can resurrect stale data

A reader misses the cache and begins fetching version 10 from the database. A writer commits version 11 and invalidates the cache. The original reader then finishes and stores version 10 after the invalidation. Future readers receive stale data even though the writer performed the expected invalidation step.

Caching is a consistency protocol as well as a performance optimization. It creates a derived representation whose lifetime and freshness must be defined. The same principle applies to materialized views, search indexes, analytical copies, and client-side state.

Cache-aside and update strategies

In cache-aside, the application checks the cache, loads on a miss, and fills the result. It is simple but exposes races between fills and writes. Time-to-live limits duration under ordinary conditions but does not establish immediate read-your-writes or prevent every stale response.

Write-through updates the cache as part of the write path, while write-behind defers authoritative persistence. These names describe families of behavior, not complete atomicity protocols. A failure between the database and cache update must be interpreted. If both are treated as authoritative, the design needs a cross-system contract or a single source of truth.

Versioned entries can prevent an older fill from replacing a newer known version if updates are ordered and comparison is safe. Invalidation generations or lease-based fills provide other mechanisms. Every approach must account for cache eviction, process restart, duplicate messages, and missing events.

Hit rates and miss amplification

At an illustrative 100,000 reads per second and 99% hits, ordinary misses are 1,000 per second. That average can conceal a stampede: if a popular entry expires, many requests may all miss before one fill completes.

Single-flight coordination lets one loader perform the fetch while others await its result. It reduces duplicate work but creates a shared waiting point and needs failure handling. A loader that hangs must not block the key forever. Stale-while-revalidate can return an older value while refreshing, but only for workloads whose freshness contract allows it.

Negative caching stores absence. It can reduce repeated misses for nonexistent objects but delay discovery after creation. Authorization failures should be handled carefully because a cached denial or result may depend on identity and policy version. Cache keys must include all relevant dimensions.

Materialized views

A materialized view stores the result of a query or a representation used to produce it. Refresh can be complete, periodic, or incremental. The maintenance policy determines freshness and write overhead.

For a grouped sum, an insertion can add its contribution to the corresponding group. A deletion subtracts the old contribution. An update can require both removing the old group contribution and adding the new one, especially if its grouping key changes. Null and aggregate semantics remain important.

For this first sketch, qualifying payments have non-null integer amounts, and count means COUNT(amount). Supporting nullable amounts and preserving empty versus all-null groups requires the additional state explained below.

on committed payment change(old, new):
    if old exists and old qualifies:
        total[old.country] -= old.amount
        count[old.country] -= 1
    if new exists and new qualifies:
        total[new.country] += new.amount
        count[new.country] += 1

This illustrates delta maintenance, not a complete fault-tolerant consumer. Duplicate or out-of-order delivery can corrupt totals unless transaction identity, sequence, or an idempotent maintenance protocol handles it. A database-integrated maintenance path can share transaction machinery, while an external consumer must establish its own atomic progress boundary.

Incremental joins and dependencies

A small input change does not always create a small output change. In a join, inserting one row can match many rows on the other side. Updating a popular dimension value can affect millions of result rows. Incremental evaluation avoids unnecessary recomputation, but output size and dependency fan-out remain real costs.

An engine can analyze declarative query expressions and derive maintenance plans. Arbitrary procedural code is harder to transform. Dependency tracking can record which rows or ranges a function read and rerun it when those inputs change. That may be much cheaper than rerunning on every transaction, while still repeating the whole function for a relevant invalidation.

SpacetimeDB’s view documentation distinguishes incremental query evaluation from procedural view reevaluation and describes sharing caller-independent view results. Convex documents reactive queries driven by tracked dependencies. These mechanisms reduce application bookkeeping while retaining server-side computation and tracking costs.

CDC and progress boundaries

Change-data capture exposes committed changes for downstream processing. A consumer needs a position or watermark identifying how far it has processed. Updating derived state and recording progress should be atomic within the consumer’s destination where possible, or use another safe replay strategy.

If a consumer records progress before applying a change, a crash can lose the change. If it applies first and records progress later, a crash can replay it. Idempotency or an atomic state-plus-offset update resolves the latter pattern. A transport’s delivery label alone does not make arbitrary external effects exactly once.

Snapshots and logs must align. A consumer taking a snapshot while changes continue needs a known boundary and catch-up sequence. Starting a stream after an unrelated snapshot can miss changes or apply an inconsistent mixture. Lesson 33 develops the client-delivery version of this problem.

CQRS and separate read models

Command-query responsibility segregation, or CQRS, separates write-side decisions from read-side representations. It can make a read model specialized and efficient. It also introduces propagation delay, replay, schema evolution, and operational dependencies.

A user completing a command may need an acknowledgment that its read model has caught up, or the interface may show a pending state. Returning success and immediately querying an asynchronous projection without explaining staleness can produce a confusing experience. A version token can connect the command’s position to a required read watermark.

The write model still owns invariants. A stale read model should not independently authorize an exclusive sale unless the architecture deliberately allocates rights to it. Using a projection for display and using it to make current authoritative decisions are different contracts.

Retention, rebuilding, and evolution

A derived representation can often be rebuilt, but rebuild time and required source history matter. If the source log has been truncated, replay alone may be insufficient. A new consumer may need a consistent snapshot plus subsequent changes.

Changing projection logic can require versioned consumers or a parallel rebuild. Reusing old derived state under incompatible semantics can mix definitions. The system should track which code or schema produced the state and when it is safe to cut over.

Rebuilds compete with live traffic. Reading the entire source and writing a new projection can consume substantial resources. A rollout plan needs rate limits, validation, and a way to compare old and new results without treating the old representation as infallible.

Solve the stale-fill race with a retained generation

Return to the reader loading version 10 while a writer commits version 11. A simple invalidation deletes the cache entry, but deletion loses the evidence that an older loader is still in flight. If the loader later performs an unconditional put, stale state returns. The design needs a coordination fact that survives long enough to reject that fill.

One conceptual local cache stores a per-key generation separate from the cached value. A loader reads generation g, fetches from the authority, then fills only if the generation remains g under an atomic compare-and-install operation. A writer advances the generation after committing the new authoritative state and invalidates the old value. The loader’s generation check and fill must be one synchronized decision.

load(key):
    generation = cache_generation(key)
    value, authoritative_version = fetch_from_authority(key)
    install_only_if_generation_unchanged(key, generation, value)
    return value under the caller's stated freshness contract

invalidate_after_commit(key):
    atomically_advance_generation_and_remove_value(key)

The sketch does not by itself provide linearizable reads. A reader already fetching old state can return that state while overlapping the write, and a gap between authoritative commit and invalidation can expose stale cache entries. It specifically prevents the late-fill resurrection under a defined local generation protocol. Stronger contracts require stronger synchronization or validation against the authority.

Generation metadata cannot be evicted arbitrarily while older loaders exist. Resetting the generation to zero can recreate an ABA-style problem in which an old token appears valid again. A bounded loader lifetime, unique epochs, or another safe retention scheme must preserve the rejection evidence. Distributed caches also need to handle lost invalidations and process restarts.

A versioned cache entry needs a meaningful version order

A monotonic authoritative version can reject an incoming value older than the retained one. An invalidation can carry a minimum version rather than merely remove data. The cache then refuses fills below that floor, including fills arriving out of order. The floor itself is correctness metadata and needs retention and replication appropriate to the contract.

A timestamp from the loader’s local wall clock is not an authoritative version. Clock skew can label old data newer than a valid update. Versions should correspond to the source’s ordering or use a comparison rule that recognizes concurrency. Independent per-key versions also cannot be compared as though they formed one global transaction order.

Negative caching needs equivalent care. A cached “not found” can outlive a newly created row. If creation advances a generation or version floor, the stale absence can be rejected or invalidated. Otherwise the service offers a TTL-bounded discovery delay rather than immediate read-your-writes. The interface should reflect that choice.

Authorization adds dimensions to the cache key and invalidation rules. A result may depend on tenant, caller identity, role membership, policy version, locale, and query parameters. Omitting any relevant dependency can expose data or return semantically wrong results. A cache hit is safe only when it represents the same effective query and permission context.

Stampedes are burst problems hidden by hit ratios

At 100,000 reads per second with 99% ordinary hits, the baseline miss load is 1,000 reads per second. Suppose one popular item receives 20,000 requests per second and expires while its source query takes fifty milliseconds. About one thousand requests can arrive during one load interval under a steady-rate approximation. If each triggers the same query, the transient miss load overwhelms the baseline calculation.

Single-flight coordination makes one loader responsible while others await a shared future. The future needs a timeout, cancellation policy, and cleanup if the loader fails. A caller timing out should not necessarily cancel a load still useful to other callers. Reference ownership and shared deadlines require deliberate design.

Stale-while-revalidate can return a previous value while one background load refreshes it. This is suitable for a catalog description but may be wrong for current stock used to authorize a sale. Freshness is data- and operation-specific: a stale display can be acceptable while the final purchase revalidates against the authority.

Randomized expiry spreads refresh work across keys, while proactive refresh can reduce misses for hot objects. Both spend capacity in anticipation of future reads. A cache should bound concurrent source loads so a cold restart does not unleash every client directly on the database. Its failure mode should degrade under control rather than multiply demand.

Delta maintenance needs old and new contributions

For a revenue-by-country view, an update can change amount, country, qualification status, or all three. Maintenance first removes the old qualifying contribution and then adds the new qualifying contribution. If a payment moves from France to Germany, modifying only Germany’s total leaves France overstated.

Assume France’s total is 1,000 cents across five qualifying rows and Germany’s is 700 across three. A payment formerly contributing 200 cents to France changes to 250 cents in Germany. The correct new states are France (800,4) and Germany (950,4). A final average uses sum and count, not a stored average adjusted by an arbitrary delta.

Null behavior affects the state. COUNT(*) includes rows with null amounts, while COUNT(amount) does not. SUM(amount) ignores nulls and generally returns null for no non-null contributions. An implementation storing only a numeric sum needs enough additional state to distinguish an empty group or all-null inputs from a genuine zero sum.

committed source transaction
    -> ordered change records with old/new values
    -> update derived groups
    -> atomically record processed source position
    -> expose projection at that progress watermark

The diagram’s atomic boundary is at the destination’s state-plus-progress update when the destination supports it. Applying changes and recording progress separately creates a replay or loss window. A transport promising at-least-once delivery does not remove that destination obligation.

Incremental joins can have large deltas

For a join Orders JOIN Customers, inserting one order usually touches one customer match if the key is unique. Updating one customer attribute can affect every joined order for that customer. A single source row can therefore produce millions of changed output rows. Incremental maintenance reduces redundant recomputation but cannot avoid producing a genuinely large explicit delta.

With simultaneous changes to both join inputs, delta algebra must include cross terms. For relations R and S, the insertion-only result difference includes deltaR JOIN S, R JOIN deltaS, and deltaR JOIN deltaS under the chosen before/after convention. Omitting the cross term misses matches where both rows are newly added. Using an inconsistent convention can instead double count them.

Deletions and duplicate rows require multiplicities. A bag-semantics view may retain counts for logically identical tuples so removing one duplicate does not remove all copies. Outer joins add unmatched-row transitions when the first match appears or the last disappears. Query semantics determine maintenance complexity, which is why arbitrary procedural code is harder to incrementalize automatically.

Dependency-based reexecution offers another point in the design space. A function records the keys or ranges it read and reruns when relevant data changes. It can avoid irrelevant reruns without deriving a small algebraic delta for every operation. Broad dependencies and expensive functions still create large recomputation work. The maintenance mechanism should be described precisely rather than calling every reactive result incremental in the same sense.

Two clients subscribing to the same caller-independent region view may share one maintained result and receive separate deliveries. Two personalized inventory views can depend on different identity contexts and require separate results. Sharing based only on query text could expose one user’s rows to another.

Even when results are shareable, fan-out remains. One computed ten-kilobyte delta delivered to ten thousand clients represents roughly one hundred megabytes of payload before framing and retransmission. Computation sharing reduces server evaluation, while interest management and delivery control address network cost. These are separate optimization dimensions.

SpacetimeDB’s view documentation distinguishes query-engine incremental maintenance from procedural reevaluation and discusses sharing caller-independent results. Convex documents dependency-tracked reactive queries. Those implementation facts motivate the comparison; the numerical scenarios here are independent teaching models rather than measured behavior of either service.

Security changes can invalidate shareability. If a player’s role is revoked, removing the subscription prevents future delivery but does not retract bytes already sent. A client may need to discard local rows, while the server must enforce authorization before further output. A cached maintained result should not be considered safe solely because its original subscription was valid.

CDC progress is a transactional relationship

Suppose a consumer applies change 501 to its derived total, crashes, and restarts from stored offset 500. Reapplying 501 doubles the delta. If it stores 501 first and crashes before changing the total, the update is lost. A transaction containing both derived mutation and offset advancement resolves the race within one destination database.

When the destination cannot atomically include the offset, another strategy can store idempotent per-source-operation effects or reconstruct complete replacement state keyed by source version. The choice depends on storage semantics and output size. Sending a non-idempotent external email remains a different boundary even if the database projection is exactly-once within its own transaction.

Source positions may be per-partition rather than globally ordered. A single scalar “processed through 501” is then insufficient to describe all progress. A vector of offsets or a source-provided global watermark can express the actual boundary. A read model’s freshness claim needs to match that position’s scope.

A source transaction affecting several partitions can also need transaction-complete delivery before the projection exposes an atomic result. Applying its changes one by one may temporarily show a broken invariant. The read model can buffer a complete transaction or explicitly provide weaker intermediate visibility. This is a deliberate contract, not an incidental detail of event handling.

Rebuilds need a snapshot-and-catch-up protocol

A projection rebuild starts from a coherent source snapshot at S and then applies every required committed change after S. The source must retain that suffix while the snapshot is read. Beginning to stream only after an unrelated full scan risks a gap; streaming first without reconciling the snapshot boundary can double count or mix versions.

A new projection generation can be built beside the old one. Validation compares selected aggregates, row counts, sampled keys, and invariant checks while recognizing that the old projection may itself contain defects. Once the new generation reaches a chosen watermark and passes checks, routing switches through a controlled boundary. Old state remains available until rollback and reader-lifetime requirements expire.

A five-terabyte source scanned at an effective 500 MB/s takes ten thousand seconds, or about 2.78 hours, using decimal units. Concurrent changes accumulate during that time, and applying them requires surplus capacity. A rebuild rate limit protects foreground traffic but extends retained-history needs. “Rebuildable” is therefore not synonymous with “cheap to recover.”

Projection code versions belong in metadata. Changing rounding, filtering, or group identity while incrementally applying onto old state can mix incompatible definitions. A full rebuild or explicit state migration is necessary when the old state cannot be transformed safely. Derived data still requires lifecycle discipline even when it is not the ultimate source of truth.

Caching moves work across time and ownership

A hit avoids recomputing or fetching a result because earlier work produced a reusable representation. The price is retained memory, invalidation or refresh work, and a freshness contract. Materialized views and reactive query systems integrate more of that work into the platform, while application caches expose more responsibility to developers.

The design should name which state is authoritative, what changes invalidate a result, how progress is recorded, how duplicates and gaps are handled, and how the representation is rebuilt. A low-latency read is meaningful only when it returns data permitted by the application’s contract. Avoided work and maintained correctness must be evaluated together.

A stale-fill timeline makes the race visible

Reader: miss -> fetch version 10 --------------------------> late fill 10
Writer:                 commit version 11 -> invalidate
Cache: empty -------------------------------> empty ------> stale 10

Invalidation removed the value but did not cancel an older load or retain evidence of version 11. A generation check or version floor can reject the final fill under its defined synchronization protocol. A TTL merely limits how long the resurrected value remains under normal expiration behavior; it does not prevent the race.

The timeline also distinguishes overlapping reads from future reads. A reader already fetching old state might return it under a permitted contract. Persisting that old state for unrelated later readers is a separate consequence. Cache designs should specify both the current caller’s freshness and the values that may be installed for future callers, because fixing one does not automatically fix the other.

Case connection and design conclusion

PostgreSQL documents materialized views as a specific stored-result facility. SpacetimeDB and Convex document different reactive maintenance models. These examples demonstrate that cached results range from application-managed entries to engine-integrated query state, each with a particular freshness and maintenance interface.

A maintained-result design states authority, dependency tracking, progress, replay, versioning, and the cost of invalidation or recomputation. Lesson 33 follows the result beyond the server into many client copies, where delivery order, reconnects, and slow consumers become central.

Sources and further reading

Course index · Lesson plan