When I interview for remote roles or sketch an architecture for a client, the same three building blocks appear every time: a cache, a queue and a rate limiter. Know them properly and you can reason about most systems, from a WordPress site surviving a traffic spike to an AI pipeline processing thousands of documents. This post is the version of those fundamentals I wish someone had given me at the start, with code instead of hand-waving.
The shape of a scalable request
1. Caching
Patterns
| Pattern | How | Use when | Cost |
|---|---|---|---|
| Cache-aside (lazy) | App checks cache; on miss reads DB and writes cache | Default. Read-heavy, tolerates brief staleness | First read is slow; stale until TTL or invalidation |
| Write-through | Every write updates DB and cache together | Reads must see writes immediately | Slower writes; caches data that may never be read |
| Write-behind | Write to cache, flush to DB asynchronously | Very high write rates (counters, analytics) | Data loss window if cache dies |
| Read-through | Cache itself loads from DB on miss | You have a caching library/proxy that supports it | Same as cache-aside, less app code |
Cache-aside, done properly
The naive version has a bug that only appears under load: when a hot key expires, a thousand concurrent requests all miss, all hit the database, and all write the cache. That is a cache stampede, and it is how a cache makes an outage worse. The fix is a short lock so only one request recomputes while the others wait or serve stale.
// lib/cache.ts -- cache-aside with stampede protection and stale-while-revalidate
import { redis } from './redis';
export async function cached<T>(key: string, ttlSec: number, loader: () => Promise<T>, staleSec = 30): Promise<T> {
const raw = await redis.get(key);
if (raw) {
const { v, exp } = JSON.parse(raw) as { v: T; exp: number };
if (Date.now() < exp) return v; // fresh
// Stale: try to become the single refresher; everyone else gets the stale value immediately.
const lock = await redis.set(`${key}:lock`, '1', 'NX', 'PX', 5000);
if (!lock) return v;
refresh(key, ttlSec, staleSec, loader).catch(() => {}); // background refresh
return v;
}
// Cold miss: one loader runs, others wait briefly then retry.
const lock = await redis.set(`${key}:lock`, '1', 'NX', 'PX', 5000);
if (!lock) { await sleep(50); return cached(key, ttlSec, loader, staleSec); }
return refresh(key, ttlSec, staleSec, loader);
}
async function refresh<T>(key: string, ttlSec: number, staleSec: number, loader: () => Promise<T>): Promise<T> {
try {
const v = await loader();
await redis.set(key, JSON.stringify({ v, exp: Date.now() + ttlSec * 1000 }), 'EX', ttlSec + staleSec);
return v;
} finally {
await redis.del(`${key}:lock`);
}
}Invalidation on write is a redis.del(key) (or a tag-based purge if you cache many keys per entity). The WordPress and Node specifics, including full-page and fragment caching, are in Redis Caching Strategies; the framework-level version is Next.js caching.
What to cache, and what not to
- Yes: rendered pages for anonymous users, product/category listings, config and settings, expensive aggregates, external API responses, query embeddings.
- No: anything per-user that changes often, anything where a stale read is a correctness bug (stock at checkout, balances), or data cheaper to compute than to serialise.
2. Queues
A queue decouples the moment work is requested from the moment it is done. The HTTP request enqueues a job and returns in 5 ms; a worker does the 3-second email send or 30-second PDF render whenever it gets to it. Three properties make queues powerful and dangerous:
- Buffering: a spike of 10,000 signups becomes a backlog processed at whatever rate your workers manage. The site stays up.
- Retries: a failed job runs again with back-off. The consumer must be idempotent, because "at least once" delivery means duplicates.
- Back-pressure: if producers outrun consumers forever, the queue grows without bound. You need a policy: bounded queue and reject, shed low-priority work, or autoscale workers on queue depth.
// Producer (inside the HTTP handler): fast, durable enqueue
await queue.add('send-welcome-email', { userId }, {
jobId: `welcome:${userId}`, // dedupe: same job enqueued twice = one job
attempts: 5,
backoff: { type: 'exponential', delay: 2000 },
removeOnComplete: 1000,
});
// Consumer: idempotent, because retries and duplicates WILL happen
worker.process('send-welcome-email', async (job) => {
const { userId } = job.data;
const already = await sql`SELECT 1 FROM email_log WHERE user_id = ${userId} AND kind = 'welcome'`;
if (already.length) return; // duplicate delivery: no-op
await mailer.send(await buildWelcome(userId));
await sql`INSERT INTO email_log (user_id, kind) VALUES (${userId}, 'welcome')`;
});Monitor two numbers: queue depth (how far behind) and oldest job age (how late). Alert on the second; a deep queue that drains fast is fine, a shallow queue with a 40-minute-old job is not. The full implementation with dead-letter queues is in Event-Driven Architecture with Redis and BullMQ.
3. Rate limiting
Rate limiting protects your system from clients that are abusive, buggy or simply more numerous than you planned for. It also protects your wallet: on an LLM-backed endpoint, it is the difference between a bad day and a bad month (see the budgets in my agent architecture post).
Algorithms
| Algorithm | Behaviour | Trade-off |
|---|---|---|
| Fixed window | N requests per calendar minute | Simple; allows 2N at a window boundary |
| Sliding window log | Exact count over the last 60 s | Accurate; stores every timestamp (memory) |
| Sliding window counter | Weighted blend of current and previous window | Good approximation, two counters |
| Token bucket | Tokens refill at rate r; each request takes one; bucket holds b | Allows bursts up to b, smooths to r. Usually what you want |
| Leaky bucket | Requests drain at a fixed rate | Smooth output; no bursts, adds latency |
Token bucket in Redis, atomically
Two round-trips (read, then write) create a race where concurrent requests both see one token. A Lua script runs atomically on the server:
-- token_bucket.lua KEYS[1]=bucket key ARGV: rate(tokens/sec), burst, now_ms, cost
local rate, burst, now, cost = tonumber(ARGV[1]), tonumber(ARGV[2]), tonumber(ARGV[3]), tonumber(ARGV[4])
local data = redis.call('HMGET', KEYS[1], 'tokens', 'ts')
local tokens = tonumber(data[1]) or burst
local ts = tonumber(data[2]) or now
local elapsed = math.max(0, now - ts) / 1000
tokens = math.min(burst, tokens + elapsed * rate) -- refill
local allowed = 0
if tokens >= cost then tokens = tokens - cost; allowed = 1 end
redis.call('HSET', KEYS[1], 'tokens', tokens, 'ts', now)
redis.call('PEXPIRE', KEYS[1], math.ceil(burst / rate * 1000) + 1000) -- self-cleaning
return { allowed, tokens }// middleware/rate-limit.ts
const script = await fs.readFile('token_bucket.lua', 'utf8');
export function rateLimit({ rate, burst, keyFor }: { rate: number; burst: number; keyFor: (req: Request) => string }) {
return async (req: Request, res: Response, next: NextFunction) => {
const [allowed, remaining] = await redis.eval(script, 1, `rl:${keyFor(req)}`, rate, burst, Date.now(), 1) as [number, number];
res.set('RateLimit-Limit', String(burst));
res.set('RateLimit-Remaining', String(Math.floor(remaining)));
if (!allowed) { res.set('Retry-After', String(Math.ceil(1 / rate))); return res.status(429).json({ title: 'Too many requests' }); }
next();
};
}
// 10 requests/sec sustained, bursts of 30, per authenticated user (fall back to IP)
app.use('/api', rateLimit({ rate: 10, burst: 30, keyFor: req => req.user?.id ?? req.ip }));Limit by the identity that matters: user ID for authenticated routes, API key for partners, IP only as a fallback (shared IPs behind NAT will punish innocent users). Layer limits: a generous per-user limit, a tighter per-endpoint limit on expensive routes, and a global limit that protects the database no matter what.
Designing for the failure modes you just added
Each block introduces a new dependency, usually Redis. Decide up front what happens when it is unavailable:
- Cache down: fall through to the database with a circuit breaker so you do not hammer it with timeouts. The site gets slower, not broken.
- Queue down: either fail the request (if the job is essential) or write the job to a database outbox table and let a worker drain it later. Never silently drop.
- Rate limiter down: fail open with a lower in-memory limit per process. Failing closed turns a Redis blip into a full outage.
// Fail-open wrapper: if Redis errors or is slow, allow the request but log it
async function safeLimit(fn: () => Promise<boolean>): Promise<boolean> {
try { return await Promise.race([fn(), sleep(50).then(() => true)]); }
catch (e) { log.warn({ e }, 'rate limiter unavailable; failing open'); return true; }
}Putting it together: a traffic spike on a content site
A client's article was picked up by a national news site; traffic went from 2,000 to 180,000 visitors in an hour. What kept it up: Cloudflare cached the HTML for anonymous visitors (cache), the newsletter signups from the article went into a queue and were processed over the next twenty minutes (queue), and a per-IP limit on the comment endpoint absorbed a bot that showed up with the crowd (rate limit). The origin server's CPU never crossed 40%. None of that is exotic; it is these three ideas applied deliberately, and it is the same approach whether the origin is Nginx and PHP or Node.
If you are preparing for system design interviews, this trio plus indexing and service boundaries covers the majority of questions; my interview prep guide has the template I use to structure the answer.