System Design - Caching

In distributed system design, caching is the single most effective technique for reducing read latency, absorbing traffic spikes, and shielding relational databases from overload. Caching stores copies of data in high-speed volatile storage (typically memory like Redis or Memcached) or edge networks (CDNs).

However, caching introduces distributed state, eventual consistency challenges, and complex invalidation dynamics. As Phil Karlton famously quipped: "There are only two hard things in Computer Science: cache invalidation and naming things."

Caching Access Patterns

The relationship between application code, the cache layer, and the underlying database defines the caching strategy.

1. Cache-Aside (Lazy Loading)

The application coordinates directly between the cache and the database:

Read Path: The application checks the cache first. If found (Cache Hit), it returns the data. If missing (Cache Miss), the application fetches from the database, writes the result to the cache with an expiration (TTL), and returns the response.
Write Path: The application writes directly to the database, then invalidates (deletes) or updates the corresponding cache key.
Pros: Resilient to cache crashes (requests fall back to the database); only requested data is cached.
Cons: Cache miss penalty (two network round-trips: cache miss + DB query); potential for stale data between updates.

2. Read-Through Cache

The application treats the cache as the primary data store:

The application requests data from the cache library/service.
On a cache miss, the cache provider itself fetches the record from the database, populates the cache entry, and returns it to the application.
Use case: Simplifies application logic; commonly implemented via database middleware or specialized object-relational caching layers.

3. Write-Through Cache

Data is written to the cache and the database synchronously in a single transaction:

The application writes an entry to the cache.
The cache immediately writes synchronously to the backing database before returning success.
Pros: No stale reads; cache is always in sync with persistent storage.
Cons: Higher write latency because every write must complete against two storage engines sequentially.

4. Write-Around Cache

Data is written directly to persistent storage, completely bypassing the cache:

Only subsequent reads will populate the cache on demand (similar to Cache-Aside).
Pros: Prevents write-heavy, infrequently-read data (e.g., real-time audit logs or bulk data uploads) from polluting the cache and evicting hot keys.

5. Write-Back (Write-Behind) Cache

The application writes directly to the cache, which immediately acknowledges success. The cache asynchronously batches and flushes dirty writes to the database in the background:

Pros: Incredible write performance and write throughput; absorbs bursts and consolidates multiple updates to the same record before hitting the DB.
Cons: Risk of data loss if the cache node crashes before flushing dirty pages to persistent storage; high implementation complexity.

The Three Inevitable Cache Disasters

During high-concurrency system design interviews, demonstrating mastery over these three failure scenarios separates junior from senior engineers:

1. Cache Avalanche (缓存雪崩)

The Problem: A massive number of cached keys expire at the exact same moment, or the entire cache cluster restarts. Tens of thousands of concurrent requests miss simultaneously, inundating the database and causing cascading outages.

Solutions:

Jittered Expiration (Random TTL): Add a random uniform duration (e.g., TTL = 3600 + rand(0, 300) seconds) to cache keys so expirations distribute evenly over time.
High-Availability Cache Clusters: Deploy Redis Sentinel or Redis Cluster across multiple Availability Zones with master-replica replication to survive node failures.
Circuit Breakers & Rate Limiting: Place Sentinel or Hystrix circuit breakers in front of the database to shed excess load when DB latency degrades.

2. Cache Breakdown / Hotspot Invalidation (缓存击穿)

The Problem: A single ultra-popular key (e.g., a viral celebrity tweet or flash-sale product) expires or is invalidated. The instant it expires, thousands of concurrent requests discover a cache miss and rush to execute the same expensive DB query concurrently.

Solutions:

Distributed Mutex Lock: Only the first thread that acquires a lock (e.g., SET lock:key token NX PX 5000) is allowed to query the database and repopulate the cache; other threads wait and retry reading from the cache.
SingleFlight Pattern (Go / Node): Merges concurrent in-flight requests for the same key into a single shared execution, ensuring only one backend query runs while all other callers share the resulting promise/channel.
Logical Expiration (Never Truly Expire): Store a logical expiration timestamp inside the value payload. When an application sees the logical timestamp is past, it returns the slightly stale cached value immediately while spawning a background goroutine/worker to asynchronously refresh the key from the DB.

3. Cache Penetration (缓存穿透)

The Problem: Malicious attackers or buggy clients query keys that exist neither in the cache nor in the database (e.g., querying user_id = -9999 or non-existent random UUIDs). Because the database returns null, no cache entry is created, and every subsequent request hits the database directly.

Solutions:

Cache Empty / Null Objects: Store a sentinel "null" value in the cache with a short TTL (e.g., 60 seconds) so repeated queries for non-existent IDs return immediately without hitting the DB.
Bloom Filters: Place an in-memory Bloom filter (or Cuckoo filter) in front of the cache. If the Bloom filter determines a key does not exist, reject the request immediately with zero disk or DB seeks.
Input Validation: Enforce strict regex validation, type boundaries, and authenticated schema filters at the API Gateway level before requests ever touch downstream services.

Cache Invalidation and Consistency

When using the Cache-Aside pattern, how do you keep the cache consistent with the database?

Invalidate (Delete) vs. Update

Best Practice: Always Delete the Cache Key, Do Not Update It.
If two concurrent writes occur (Thread A writing x=1 and Thread B writing x=2), network jitter could cause Thread B to update the DB first, then Thread A to update the DB, while Thread A's cache update lands after Thread B's cache update, leaving permanently corrupted stale data in the cache.
Deleting the key on every write avoids this race condition; the next read will deterministically fetch the latest value.

Dual-Write Dilemma & Change Data Capture (CDC)

In high-concurrency production systems, executing "Update DB -> Delete Cache" inside application code can still fail if the application crashes before the delete executes, or if network partitions occur.

Modern scalable architectures solve this via Change Data Capture (CDC):

Application writes strictly to the database.
The database commits the change to its Write-Ahead Log (e.g., MySQL binlog or PostgreSQL wal).
A background tool (such as Debezium, Canal, or an engine like Kafka Connect) tails the database binlog sequentially.
The consumer asynchronously sends cache invalidation events to Redis.
This decouples business logic from cache maintenance and guarantees eventual consistency even during application restarts.

Cache Eviction Policies

When cache memory fills up to its configured ceiling (maxmemory), the engine must evict existing keys to make room for new ones:

Eviction Policy	Mechanism	Best Suited For
LRU (Least Recently Used)	Evicts keys that have not been read for the longest time	General web applications; standard Pareto 80/20 access patterns
LFU (Least Frequently Used)	Tracks access counters; evicts keys with the lowest hit frequency	Long-running services where access frequency matters more than recency
FIFO (First In, First Out)	Evicts keys in the order they were inserted	Time-series queues where older data naturally becomes obsolete
Random Eviction	Evicts random keys uniformly	Homogeneous workloads where all keys have equal access probabilities
TTL-Based Eviction (`volatile-ttl`)	Evicts keys with the shortest remaining time-to-live	Workloads where urgent expiration takes precedence over access history

Tiered Caching Architecture

Production systems rarely rely on a single cache:

[ Client Browser Cache ] ──► [ CDN (Cloudflare/CloudFront) ]
                                            │
[ Redis In-Memory Cluster ] ◄── [ API Gateway / In-Process L1 Cache ]
            │
[ Database Buffer Pool (InnoDB / Postgres) ]

L1 In-Process Cache: Fast, zero-network in-memory cache directly inside the application process (e.g., Go sync.Map, Java Caffeine). Nanosecond access latency.
L2 Distributed Cache: Shared centralized cluster (e.g., Redis Cluster, Memcached). Sub-millisecond latency; survives application instance restarts and scaling events.
Edge CDN: Geographically distributed edge servers caching static assets and precomputed JSON fragments close to global users.