The Ready Chai Counter: Caching Explained Without Confusion
SnackNow keeps asking its database for the same menu. Aman builds a ready-chai counter to explain cache hits, misses, TTL, invalidation, eviction, and stampede protection.
Caching works when SnackNow keeps reusable answers close without forgetting where the truth lives.
SnackNow Keeps Asking the Same Question
Aman's performance traces reveal something almost embarrassing in its simplicity. At 7 PM, thousands of users open SnackNow and ask for today's menu. Masala chai. Samosa. Momos. Sandwich combo. The answer barely changes during the rush, yet every request travels to the database as if nobody has asked before.
The database reads the same menu rows, joins the same price and availability data, converts the same result, and sends it back. Then the next user arrives and the entire conversation repeats. Riya waits, Meera sees the database CPU climb, and Aman watches menu latency pull the rest of the app into trouble.
Aman: 'Why are we cooking the same answer again and again?'
This is the kind of problem caching is built to solve: repeated reads where a reusable answer is cheaper to keep nearby than to rebuild from the source every time.
Caching is valuable when repeated work is expensive, the answer is reusable, and the product can define how fresh that answer must be.
What Is Caching? The Ready Chai Counter
Meera would never send every chai customer into the kitchen to watch water boil. During the rush, she keeps ready chai near the counter. A customer receives it quickly, the kitchen avoids repeating the complete process for every cup, and the counter is replenished when needed.
A software cache follows the same idea. It stores a copy of data or a computed result in a faster layer, often memory or a nearby edge server. Future requests can read that copy instead of repeating a database query, network call, file read, or expensive calculation.
The cache is usually not the source of truth. The database, object store, or owning service remains authoritative. The cache is a faster copy with a freshness and removal policy.
That distinction prevents a dangerous mental shortcut. If SnackNow loses its menu cache, the app should become slower while it refills, not forget what the menu is. If losing the cache destroys the only copy, it was acting as primary storage, not merely a cache.
Visual purpose: This visual helps you understand how caching changes the path and removes repeated database work.
How to read this visual: compare the two sides. Without caching, every arrow reaches the database. With caching, repeated reads stop at the faster layer and only misses continue to the source.
Checkpoint: A cache trades extra state and freshness complexity for lower latency and less work on the slower source.
Cache Hit and Cache Miss
Riya opens the menu. SnackNow asks the cache for the key menu:today. If the key exists and is still valid, the cache returns the value. That is a cache hit. The database is not involved in the read path.
If the key is absent, expired, or intentionally invalidated, SnackNow has a cache miss. The application must fetch the menu from the database, return it, and often place a copy in the cache so the next request can hit.
Aman watches hit ratio because it tells him how often the cache satisfies attempted reads. One common calculation is hits divided by hits plus misses. A high ratio is useful only if the cached answers are correct and the cache cost is justified. A 99% hit ratio on data nobody needed to cache is not a victory.
Outcome | What SnackNow does | User effect | System effect |
|---|---|---|---|
Hit | Return the valid cached menu | Fast response | Database work avoided |
Cold miss | Fetch from database and populate cache | First request waits longer | Future reads may become hits |
Expired miss | Treat the old entry as unavailable | Request waits for refresh | Fresh copy is loaded |
Invalidated miss | Reload after a known update | Fresh price is returned | Correctness is favored |
Takeaway: misses are normal. The design question is whether misses are controlled, affordable, and safe under concurrency.
Where Caches Live
Caching is not one Redis box placed beside every architecture diagram. The best location depends on what is being reused and who can safely share it.
Visual purpose: This visual helps you understand which cache location removes which kind of repeated work and what new risk it introduces.
Cache type | Lives where | SnackNow example | Best for | Main risk |
|---|---|---|---|---|
Client/browser | User device | Menu images and static assets | Avoiding repeat downloads | Old assets or private data on device |
CDN/edge | Near users across regions | Food images, CSS, public menu payloads | Reducing origin latency and bandwidth | Serving stale or wrongly shared responses |
Application local | Inside one app instance | Small configuration lookup | Very fast instance-local reads | Different instances hold different copies |
Distributed server cache | Shared cache service | Today's menu in Redis or Memcached | Shared hot data across app instances | Network dependency and invalidation complexity |
Database-side | Database engine or stored summary | Precomputed sales result | Repeated expensive data access | Refresh and write overhead |
How to read this visual: move from the user toward the source. Caches closer to the user remove more network work, while shared server caches make one reusable answer available across application instances. Each location has a different sharing boundary.
Redis is popular for distributed caching because it keeps data in memory, supports expiration, rich data structures, replication, and clustering. Memcached is a simpler distributed in-memory cache. Hazelcast and Apache Ignite provide broader distributed data-grid capabilities. Aman chooses by required behavior, operational skill, consistency expectations, and cost - not by tool popularity.
Cache-Aside: Fetch Only When Needed
SnackNow uses cache-aside, also called lazy loading, for the menu. The application owns the flow. It checks the cache first. On a hit, it returns. On a miss, it reads the database, stores the result in the cache with a suitable expiration, and returns the result.
This pattern is efficient when only a subset of data is hot. SnackNow does not need to preload every restaurant menu in every city. Popular menus enter the cache because users ask for them. Quiet menus stay in the database until needed.
Visual purpose: This visual helps you understand the complete cache-aside read path, including where a hit returns and where a miss populates the cache.
How to read this visual: follow the cache check first. The hit branch ends immediately. Only the miss branch reaches the database, and that branch writes the result into the cache before returning.
Build a stable cache key that includes every dimension that changes the answer, such as restaurant, date, region, and language.
Read the cache and return immediately on a valid hit.
On a miss, read the source of truth.
Store only the reusable result with a deliberate TTL.
Return the source result even if populating the cache fails, when the product can safely degrade that way.
The last step matters. If Redis is temporarily unavailable, SnackNow should often fall back to the database with protection rather than turn a performance optimization into a complete outage. The database still needs enough capacity and rate protection to survive that degraded mode.
Common mistake: A cache key that forgets user, region, currency, or authorization context can return the wrong answer even when the cache is technically working.
Write-Through vs Write-Back
Reads are only half the story. Meera changes the samosa price. Now SnackNow must decide how writes reach the cache and the database.
With write-through, the write path updates durable storage and the cache as part of the same logical operation before success is reported. Reads are more likely to find fresh cached data, but writes do additional work and may wait on both systems. The exact ordering and failure handling still require design; the phrase does not magically create a distributed transaction.
With write-back, also called write-behind, the cache accepts the change first and durable storage is updated asynchronously. The write can feel fast, but a cache failure before persistence can lose data. The design also needs ordering, retries, durability, and a way to recover pending writes.
How is it so far?
Vote with other readers
Visual purpose: This visual helps you understand the consistency, latency, and durability tradeoff between two cache write strategies.
Strategy | Write path | Strength | Risk | SnackNow fit |
|---|---|---|---|---|
Write-through | Update durable store and cache before success | Fresher reads and simpler mental model | Slower writes and partial-failure handling | Menu and profile changes when freshness matters |
Write-back | Update cache, persist later | Very fast writes and batching potential | Data loss, ordering, and replay complexity | Only carefully designed, recoverable workloads |
Database then invalidate | Update source, delete cached copy | Common and straightforward with cache-aside | Short race windows and next-read miss | SnackNow menu price updates |
How to read this visual: choose by the write guarantee, not just speed. If losing a recent write is unacceptable, write-back needs durable buffering and recovery before it is safe.
A fast cache write is not a successful business write until the system can explain durability and failure recovery.
TTL and Eviction: The Counter Has Limited Space
The ready-chai counter has two limits. Chai cannot sit there forever, and the counter cannot hold every possible item. Caches face the same two pressures: freshness over time and finite memory.
TTL, or time to live, expires an entry after a chosen duration. A five-minute menu TTL means the cached copy is eligible to disappear after five minutes. TTL is a freshness backstop, not proof that the value was correct for all five minutes.
Eviction removes entries when the cache needs space. LRU removes the least recently used item. LFU removes an item used least frequently. FIFO removes the oldest inserted item. Real products such as Redis expose specific policies and approximations, so Aman must match the configured policy to actual access patterns instead of assuming the textbook label is exact.
Visual purpose: This visual helps you understand the difference between an entry expiring because time passed and being evicted because memory is full.
How to read this visual: the top lane follows one key toward TTL expiry. The bottom lane shows a full cache choosing a victim based on access or insertion history. Expiration and eviction can both create misses, but for different reasons.
Policy | Removes | Good fit | Watch out for |
|---|---|---|---|
TTL | Entries whose freshness window ended | Data with a known acceptable age | Synchronized expiries and stale windows |
LRU | Least recently accessed entries | Hot recent working sets | One-time scans can disturb recency |
LFU | Least frequently accessed entries | Stable repeatedly popular keys | Old popularity needs decay |
FIFO | Oldest inserted entries | Simple age-based capacity policy | Frequently used old entries may be removed |
Takeaway: TTL answers 'how long may this copy live?' Eviction answers 'what leaves when memory is scarce?' SnackNow needs both decisions.
Cache Invalidation: The Hard Part
At 7:15 PM, Meera changes the sandwich combo from 199 to 179. The database is correct. The cache still says 199. SnackNow has made reads fast and truth confusing.
Aman can use TTL-only freshness for data where short staleness is acceptable. For explicit updates, a common cache-aside pattern is to commit the database change first and then delete the cached key. The next read misses and repopulates from the updated source. Deleting before the database commit can allow another request to miss, read the old database value, and put the stale value back.
TTL limits how long unnoticed staleness can survive.
Manual or application invalidation removes known keys after a successful source update.
Event-based invalidation lets the owning service announce that related keys must be removed or refreshed.
Change data capture can derive invalidation events from committed database changes when the architecture justifies that machinery.
Versioned keys make a new version naturally bypass an old cached value, at the cost of cleanup planning.
Invalidation should be designed before the cache ships. Aman writes down who owns the data, which keys contain it, what freshness is acceptable, what event changes it, and what happens when invalidation fails.
If the team cannot explain how a cached value becomes fresh, it has not finished designing the cache.
Cache Stampede: When Everyone Rushes the Kitchen
At exactly 7:30 PM, the popular menu key expires. Five thousand requests arrive within a second. Every request observes the same miss. Every request turns toward the database. The cache that protected the database for thirty minutes has now converted normal traffic into a synchronized surge.
This is a cache stampede, also called a thundering herd. It is especially dangerous for hot keys whose source query is expensive. The miss path must be designed for concurrency, not only correctness in a single-request diagram.
Visual purpose: This visual helps you understand how synchronized misses overload the source and how one coordinated refresher protects it.
How to read this visual: the red path shows every miss reaching the database. The protected path allows one request to refresh while others wait, share the result, or briefly use an acceptable stale copy.
A short distributed lock or mutex lets one request rebuild the value while others wait or retry briefly.
Request coalescing combines concurrent misses for the same key into one source request.
Stale-while-revalidate serves a slightly old but acceptable value while one background refresh runs.
Randomized TTL, often called jitter, spreads expirations so many hot keys do not vanish at the same instant.
Proactive refresh renews extremely hot keys shortly before expiry when the access pattern justifies it.
A cache miss path is production code. It needs capacity limits, timeouts, and concurrency protection just like the hit path.
Distributed Caching: One Counter Is Not Enough
SnackNow now runs several application instances. A local in-memory cache inside each instance is fast, but each instance can hold a different menu copy. Restarts erase warm data, memory is duplicated, and traffic distribution decides which users see which cache state.
A distributed cache gives the application fleet a shared caching layer. Keys can be partitioned across nodes, often using hashing. Replication can keep additional copies for availability. Redis Cluster provides partitioning and replication behavior; Memcached commonly distributes keys across nodes from the client side. Hazelcast and Apache Ignite offer broader distributed computing and data-grid models.
Distribution adds its own failure modes: network hops, hot keys concentrated on one shard, node movement, replica lag, connection limits, and partial availability. Consistent hashing can reduce how many keys move when nodes change, but it does not solve every balancing or replication concern.
Need | Possible direction | Why | New responsibility |
|---|---|---|---|
Tiny instance-local configuration | Local memory | Lowest access latency | Copies can drift across instances |
Shared hot application data | Redis or Memcached | One reusable layer for the fleet | Operate networked cache capacity and failure |
Large partitioned cache | Clustered distributed cache | Spread keys and traffic | Shard balance, replication, and failover |
Global static delivery | CDN | Serve near users | Cache headers, purge, and regional freshness |
The database still needs protection for cache outages. A distributed cache is a performance dependency, and if every application instance falls through at once, the source can fail faster than the cache.
When Not to Cache
Caching is not a reward for completing the architecture diagram. It should remove measured repeated work at an acceptable correctness cost.
Do not broadly cache highly sensitive or user-private data without a precise identity and authorization boundary.
Avoid caching rapidly changing payment authorization state when the next decision requires the source's latest truth.
Do not cache data that is rarely read; the lookup and invalidation machinery may cost more than the source query.
Do not cache a cheap result merely because a cache exists.
Avoid caching when no acceptable staleness window or reliable invalidation path can be defined.
Do not let a cached error live longer than the underlying failure unless negative caching is deliberately short and safe.
Sometimes the correct optimization is a better query, smaller response, index, batch, or CDN rule. Aman starts with the repeated cost he measured and chooses the simplest layer that removes it.
Common Caching Mistakes
Caching everything
Memory fills with cold values, useful keys are evicted, and the team inherits invalidation work with little benefit.
Using no TTL or one TTL for every key
Different data has different freshness and recomputation cost. Permanent keys grow stale; overly short TTLs create avoidable misses.
Ignoring invalidation until later
The first price update becomes an incident because nobody knows which key contains the old value.
Forgetting cache stampede protection
A perfect hit path hides a dangerous miss path until a hot key expires during peak traffic.
Caching user-private data under a shared key
The response can be fast and catastrophically wrong. Cache keys and cacheability rules must include the real authorization boundary.
Treating the cache as always available
When the cache fails, unrestricted database fallthrough can overload the source. Degraded behavior needs rate limits and capacity planning.
Final Caching Cheat Sheet
Visual purpose: This visual helps you understand the decisions that turn caching from a speed trick into a dependable design.
Concept | Simple meaning | SnackNow hook | Use when | Watch out for |
|---|---|---|---|---|
Cache | Faster reusable copy | Ready chai counter | Reads repeat expensive work | Freshness and extra state |
Hit | Value found | Ready chai available | Measuring useful cache reads | Correctness still matters |
Miss | Value absent | Kitchen must prepare | Loading from source | Burst load on source |
Cache-aside | App loads on demand | Check counter, then kitchen | Only some data is hot | Invalidation and race windows |
Write-through | Persist in write path | Update source and counter | Fresher read copies | Write latency and partial failure |
Write-back | Persist later | Record at counter, kitchen later | Recoverable high-speed writes | Loss and ordering risk |
TTL | Expiry time | Chai freshness window | Bounded staleness | Too short or synchronized expiry |
Eviction | Remove for space | Clear counter shelf | Finite memory | Wrong policy lowers hit ratio |
Invalidation | Remove known stale copy | Price changed | Source updates | Missing related keys |
Stampede | Many misses together | Everyone rushes kitchen | Hot-key protection | Database surge |
How to read this visual: start with the repeated read, choose where the copy lives, define the read and write pattern, then write down expiration, invalidation, miss protection, and degraded behavior.
The Mental Model Aman Keeps
Aman does not remember caching as 'put Redis in front of the database.' He remembers a ready counter with rules. What answer is worth keeping? Who may share it? Where does truth live? How old may the copy become? What removes it? What happens when everybody misses together?
For SnackNow's menu, the answer is clean: the database owns the menu, a shared cache keeps popular menu responses close, cache-aside loads them on demand, TTL bounds unnoticed staleness, application updates invalidate known keys, and stampede protection keeps one expiry from becoming a database incident.
For payment state, the answer may be different because freshness and correctness dominate. That is not a failure of caching. It is evidence that Aman is choosing by data semantics instead of applying one pattern everywhere.
A good cache makes repeated reads cheaper without making truth impossible to find.
Lock in the takeaway
Frequently asked questions
What is caching in system design?
Caching stores a reusable copy of frequently accessed or expensive data in a faster layer so later requests can avoid repeating slower work. The primary database or service usually remains the source of truth.
What is a cache hit and a cache miss?
A cache hit occurs when the requested value is available in the cache. A cache miss occurs when it is absent or expired, so the application must fetch or compute the value from the source before responding.
How does the cache-aside pattern work?
The application checks the cache first. On a miss, it reads from the database, stores the result in the cache with an expiration policy, and returns it. Updates normally change the source and invalidate the cached copy.
What is the difference between write-through and write-back caching?
Write-through updates the cache and durable store in the write path, favoring consistency with additional write latency. Write-back acknowledges the cache update first and persists later, improving write speed but adding durability and ordering risk.
What is TTL in caching?
TTL, or time to live, is the period after which a cached item expires. It limits staleness and memory use, but a TTL that is too short creates misses while one that is too long can serve outdated data.
What causes a cache stampede?
A cache stampede happens when many requests miss the same popular key at once and all fall through to the database. Locks, request coalescing, stale-while-revalidate, and randomized expiration can reduce the burst.
When should data not be cached?
Avoid or tightly scope caching when data is highly sensitive, must be strictly fresh, changes faster than it can be invalidated, is rarely read, or is cheaper to fetch than the added cache complexity.
