8.6 — Infrastructure & multi‑region¶
Module 8 · Lesson 6 · 🔴 Advanced · ~40 min
What you'll learn¶
- Where to run compute, and why "the edge" is usually the wrong answer for commerce
- Multi‑region architecture with read replicas, and the consistency problems it creates
- Node.js tuning for SSR workloads: event loop, pooling, memory
- Autoscaling for a 14× Black Friday peak
Where compute should run¶
The instinct is "as close to the user as possible". For a data‑heavy commerce render, that's wrong.
Scenario: user in Sydney, primary database in us-east-1
PDP render needs 5 backend calls.
A) Edge render in Sydney
user→edge: 20ms
5 × edge→us-east: 5 × 200ms (2 dependent rounds, 3 parallel) ≈ 400ms
TOTAL ≈ 420ms
B) Regional render in us-east-1
user→origin: 200ms
5 × local calls: ≈ 15ms (parallel)
TOTAL ≈ 215ms
C) Regional render in ap-southeast-2 with a local read replica
user→origin: 20ms
5 × local calls: ≈ 15ms
TOTAL ≈ 35ms
D) Static/ISR shell from the CDN edge in Sydney
TOTAL ≈ 20ms
The ranking: D > C > B > A. Caching beats compute placement, and compute should sit next to data, not next to users — unless you move the data too.
Edge rendering is right when the page needs no backend data, or only data already at the edge (a KV store, an edge config). For commerce pages that need catalog, pricing, and inventory, regional compute next to the data wins.
The decision table¶
| Workload | Where | Why |
|---|---|---|
| Static/ISR HTML | CDN edge | No compute needed |
| Middleware (routing, headers, A/B) | Edge | Cheap, latency‑sensitive, no data |
| PDP/PLP render | Regional Node, near the data | 5–6 backend calls |
| Cart/checkout render | Regional Node, near the transactional DB | Writes; consistency |
| Search | Regional, near the search cluster | |
| Image transforms | CDN / image service | Specialized, cacheable |
| Webhooks, cron | Regional | Not latency‑sensitive |
Multi‑region architecture¶
┌─────────────────────────┐
│ Global CDN (anycast) │
│ · static, immutable │
│ · HTML with SWR │
└───────┬──────────┬──────┘
miss │ │ miss
┌─────────────────▼──┐ ┌──▼──────────────────┐
│ us-east-1 │ │ eu-west-1 │
│ Next.js (Node) │ │ Next.js (Node) │
│ Redis (regional) │ │ Redis (regional) │
│ Postgres PRIMARY │◀───│ Postgres REPLICA │
│ Search cluster │ │ Search cluster │
└────────────────────┘ └─────────────────────┘
▲ │
└── writes routed to primary
Reads local, writes to the primary. This is the standard pattern and it works well for commerce because the read/write ratio is roughly 100:1.
// lib/db.ts
import 'server-only';
import { Pool } from 'pg';
const REGION = process.env.REGION ?? 'us-east-1';
// Reads: local replica (fast). Writes: primary (correct).
const readPool = new Pool({
connectionString: process.env[`DATABASE_REPLICA_URL_${REGION.replace(/-/g, '_').toUpperCase()}`]
?? process.env.DATABASE_URL,
max: 20,
idleTimeoutMillis: 30_000,
connectionTimeoutMillis: 2_000,
});
const writePool = new Pool({
connectionString: process.env.DATABASE_PRIMARY_URL,
max: 10,
idleTimeoutMillis: 30_000,
});
export const db = {
read: <T>(sql: string, params?: unknown[]) => readPool.query<T>(sql, params),
write: <T>(sql: string, params?: unknown[]) => writePool.query<T>(sql, params),
};
The replication lag problem¶
t=0 User in Frankfurt adds an item to their cart → write to primary (us-east-1)
t=40ms Redirect to /cart → read from eu-west-1 replica
t=40ms Replica hasn't received the write yet → EMPTY CART 💀
This is the number one bug in multi‑region commerce, and it looks like a flaky, unreproducible "items disappear from cart" report.
Read‑your‑writes, implemented properly:
// lib/db-consistency.ts
import { cookies } from 'next/headers';
const WRITE_STICKY_MS = 5_000;
/** Call after any write. Pins this user's reads to the primary briefly. */
export async function markRecentWrite() {
(await cookies()).set('rw', String(Date.now()), {
maxAge: 10, path: '/', sameSite: 'lax', httpOnly: true,
});
}
/** Use for reads that must reflect this user's own recent writes. */
export async function readConsistent<T>(sql: string, params?: unknown[]) {
const marker = (await cookies()).get('rw')?.value;
const recent = marker && Date.now() - Number(marker) < WRITE_STICKY_MS;
return recent ? db.write<T>(sql, params) : db.read<T>(sql, params);
}
Which reads need consistency:
| Read | Consistency |
|---|---|
| Product catalog | Eventual — replica is fine |
| Category listing | Eventual |
| Search | Eventual |
| Cart | Read‑your‑writes |
| Checkout | Strong — always the primary |
| Order confirmation | Strong |
| Account/profile | Read‑your‑writes |
| Inventory at checkout | Strong |
Getting this table right is the difference between a working multi‑region deployment and a stream of unreproducible bug reports.
Node.js tuning for SSR¶
The event loop is your bottleneck¶
Node is single‑threaded for JavaScript. React's renderToPipeableStream is CPU work on that thread.
Anything blocking it delays every concurrent request.
// lib/event-loop-monitor.ts
import { monitorEventLoopDelay } from 'node:perf_hooks';
const histogram = monitorEventLoopDelay({ resolution: 10 });
histogram.enable();
setInterval(() => {
metrics.gauge('event_loop_delay_p50_ms', histogram.percentile(50) / 1e6);
metrics.gauge('event_loop_delay_p99_ms', histogram.percentile(99) / 1e6);
histogram.reset();
}, 10_000);
Targets: p50 < 10 ms, p99 < 100 ms. Above that, requests are queuing and your TTFB p99 is suffering regardless of how fast your data layer is.
What blocks the event loop in a Next.js app:
| Blocker | Fix |
|---|---|
Large JSON.parse (a 5 MB API response) |
Shape responses smaller at the BFF (7.1) |
| Synchronous crypto | Use the async variants |
fs.readFileSync at request time |
Read at startup, cache in memory |
| Big array operations on request data | Move to the BFF, or paginate |
| Rendering an enormous component tree | Reduce nodes; stream |
Image processing (sharp) in‑process |
Use an image CDN (2.1) |
| Source map generation in production | Disable it |
Sizing¶
# Kubernetes: one Node process per container, CPU-request sized to ~1 core
resources:
requests: { cpu: "1000m", memory: "1Gi" }
limits: { cpu: "2000m", memory: "2Gi" }
env:
# Keep the heap under the container limit so the OOM killer never fires
- name: NODE_OPTIONS
value: "--max-old-space-size=1536"
# UV_THREADPOOL_SIZE matters if you do fs/crypto/zlib work
- name: UV_THREADPOOL_SIZE
value: "8"
Scale horizontally, not vertically. A Node process can't use 4 cores for JavaScript, so a 4‑core pod running one process wastes 3 cores. Prefer more small pods, or run a cluster of processes sized to the CPU limit.
Keep‑alive to backends¶
// instrumentation.ts — runs once at server startup
import { Agent, setGlobalDispatcher } from 'undici';
export async function register() {
setGlobalDispatcher(new Agent({
keepAliveTimeout: 30_000,
keepAliveMaxTimeout: 60_000,
connections: 128, // per origin
pipelining: 1,
}));
}
Without keep‑alive, every backend call pays a TLS handshake — 50–150 ms each, and a PDP makes 5–6. This is frequently a 200–400 ms TTFB win from four lines of configuration.
Cold starts¶
| Deployment model | Cold start | Notes |
|---|---|---|
| Long‑running containers | None | Preferred for high‑traffic commerce |
| Serverless functions | 100–800 ms | Fine for low traffic; painful at p99 |
| Edge functions | ~0–5 ms | But limited APIs and no DB pooling |
If you're on serverless:
// ❌ Every cold start pays this
import { HeavySdk } from 'heavy-sdk';
const client = new HeavySdk({ /* … */ }); // 180ms
const catalogIndex = buildIndex(require('./catalog.json')); // 400ms 💀
// ✅ Lazy and memoized: only requests needing it pay, once per instance
let clientPromise: Promise<HeavySdk> | null = null;
export function getClient() {
clientPromise ??= import('heavy-sdk').then((m) => new m.HeavySdk({ /* … */ }));
return clientPromise;
}
Also: measure your server bundle size. A 40 MB server bundle takes meaningfully longer to cold start than a 4 MB one.
Autoscaling for peak¶
Black Friday at 14× median RPS is an architecture problem, not a scaling‑config problem.
Scale on the right signal¶
# ❌ CPU-based scaling reacts too late for Node SSR — the event loop
# saturates before CPU looks alarming
metrics:
- type: Resource
resource: { name: cpu, target: { type: Utilization, averageUtilization: 70 } }
# ✅ Scale on a request-rate signal, with generous headroom
metrics:
- type: Pods
pods:
metric: { name: http_requests_per_second }
target: { type: AverageValue, averageValue: "40" }
behavior:
scaleUp:
stabilizationWindowSeconds: 0 # scale up immediately
policies: [{ type: Percent, value: 100, periodSeconds: 30 }]
scaleDown:
stabilizationWindowSeconds: 600 # scale down slowly
policies: [{ type: Percent, value: 10, periodSeconds: 60 }]
Pre‑scale, don't react¶
Autoscaling takes 30–120 seconds to add capacity. A traffic spike at a campaign email send is faster than that.
Black Friday plan:
T-48h Scale baseline to 3× normal. Verify with a load test at 5× expected peak.
T-24h Freeze deploys. Pre-warm caches: top 50K products, all categories.
T-2h Scale to 8× baseline. Confirm all pods healthy and warm.
T-0 Campaign send. Autoscaler handles the remainder.
T+6h Begin scaling down at 10% per minute.
Degrade gracefully¶
Decide in advance what to turn off under load. Make it a flag, not a code change.
// lib/load-shedding.ts
export async function getRecommendations(productId: string) {
if (await isFeatureShed('recommendations')) return []; // flag flip, instant
return recsBreaker.run(
() => fetchWithTimeout(`${RECS_API}/${productId}`, { timeoutMs: 600 }),
() => [],
);
}
Aurora's Black Friday shed list, in order:
| Priority | Feature | Effect when shed |
|---|---|---|
| 1 | Personalized recommendations | Static bestsellers instead |
| 2 | Recently viewed | Section hidden |
| 3 | Review summaries on PLP | Stars hidden on tiles |
| 4 | Live stock counts ("only 3 left") | Binary in/out of stock |
| 5 | Search typeahead | Plain search box |
| 6 | Non‑essential images (badges, lifestyle) | — |
Never shed: add to cart, cart, checkout, payment. Those are the point.
Load testing¶
Test the real path, with a realistic mix, from outside your network.
// k6/peak-load.js
import http from 'k6/http';
import { check, sleep } from 'k6';
export const options = {
stages: [
{ duration: '5m', target: 500 }, // ramp
{ duration: '10m', target: 500 }, // steady
{ duration: '3m', target: 7000 }, // 14× spike
{ duration: '10m', target: 7000 }, // sustained peak
{ duration: '5m', target: 500 }, // recovery
],
thresholds: {
'http_req_duration{page:pdp}': ['p(75)<400', 'p(99)<2000'],
'http_req_duration{page:checkout}': ['p(75)<800', 'p(99)<3000'],
http_req_failed: ['rate<0.001'],
},
};
// Traffic mix must match reality, including the cacheable/uncacheable ratio
export default function () {
const r = Math.random();
if (r < 0.31) http.get(`${__ENV.BASE}/p/${randomSku()}`, { tags: { page: 'pdp' } });
else if (r < 0.58) http.get(`${__ENV.BASE}/c/${randomCat()}`, { tags: { page: 'plp' } });
else if (r < 0.72) http.get(`${__ENV.BASE}/`, { tags: { page: 'home' } });
else if (r < 0.84) http.get(`${__ENV.BASE}/search?q=${randomQuery()}`, { tags: { page: 'search' } });
else http.get(`${__ENV.BASE}/cart`, { tags: { page: 'cart' } });
sleep(Math.random() * 3 + 1);
}
Load test findings that only show up under load:
- Connection pool exhaustion (works at 100 RPS, fails at 2,000)
- Cache stampedes when a popular entry expires under load
- Event loop saturation → TTFB p99 explosion
- Backend rate limits you didn't know existed
- Autoscaler lag
- Memory growth → OOM kills → cascading restarts
- Log volume saturating your logging pipeline (a real and common outage cause)
Run the full peak test at least twice before peak season: once 8 weeks out to find problems, once 2 weeks out to verify the fixes.
Aurora's infrastructure results¶
| Change | Effect |
|---|---|
| Keep‑alive to the BFF | TTFB −190 ms |
| eu‑west‑1 region + read replica | EU TTFB 410 → 95 ms |
| ap‑southeast‑2 region | AU TTFB 890 → 130 ms |
| Shared Redis ISR cache handler | Hit ratio 11% → 96% |
Event‑loop monitoring + JSON.parse reduction |
p99 TTFB −680 ms |
| Request‑rate autoscaling | Zero capacity incidents at peak |
| Load shedding flags | Peak survived at 16× with no checkout degradation |
The multi‑region work was the largest project in the whole programme (a quarter, three engineers) and delivered less than the caching work did. Sequence it accordingly — caching first, always.
Common mistakes¶
| Mistake | Cost |
|---|---|
| Edge rendering data‑heavy pages | Slower than regional; 5 round trips instead of 1 |
| No read‑your‑writes handling | Cart appears empty after add — the classic multi‑region bug |
| Reading from a replica at checkout | Wrong prices, wrong inventory, oversells |
| No keep‑alive | 50–150 ms per backend call |
| Vertical scaling of Node | Wasted cores |
| CPU‑based autoscaling for SSR | Reacts after the event loop is already saturated |
| Reactive scaling only | 30–120 s of degradation at every spike |
| No load shedding plan | Everything degrades equally, including checkout |
| Load testing without a realistic traffic mix | Passes the test, fails in production |
| Multi‑region before caching | Expensive project, smaller payoff |
Lab 8.6 — Infrastructure review¶
- Map compute vs data. For each route, where does it render and where does its data live? Count cross‑region round trips.
- Segment TTFB by country in RUM. Find your worst region and compute the revenue at stake.
- Check keep‑alive — is a shared agent configured? Measure a backend call's connect time.
- Add event‑loop monitoring. Chart p50/p99 for a week. Anything over 100 ms p99 needs investigation.
- Audit read consistency: list every read and classify it (eventual / read‑your‑writes / strong). Fix anything in cart or checkout reading from a replica.
- Write the load‑shedding plan with priority order and flags. Test each flag in staging.
- Run a peak load test at 1.5× expected peak with a realistic traffic mix.
Checklist¶
- Compute placed near data, not near users (unless data is replicated)
- Caching maximized before any multi‑region work
- Read/write splitting with read‑your‑writes for cart and account
- Checkout and inventory always read from the primary
- Keep‑alive configured for all backend HTTP
- Event‑loop delay monitored with alerts
- Heap size configured below the container limit
- Horizontal scaling; one process sized to its CPU limit
- Autoscaling on request rate with fast scale‑up, slow scale‑down
- Pre‑scaling plan for known traffic events
- Load‑shedding flags with a documented priority order
- Peak load test passed at 1.5× expected, twice