8.5 — Experimentation without regressions¶

Module 8 · Lesson 5 · 🔴 Advanced · ~35 min

What you'll learn¶

Why client‑side A/B testing is one of the worst things you can do to LCP and CLS
Server‑side and edge assignment patterns that cost nothing
Feature flags that don't ship dead code to every user
Measuring performance as an experiment, with proper guardrails

The client‑side A/B problem¶

The standard vendor implementation:

<!-- The "anti-flicker snippet" — near the top of <head>, blocking -->
<style>body { opacity: 0 !important; }</style>
<script src="https://ab-vendor.example.com/client.js"></script>
<script>
  // Reveal when the SDK is ready, or after a timeout
  setTimeout(() => { document.body.style.opacity = '1'; }, 4000);
</script>

What this does to your metrics:

Metric	Effect
LCP	The page is invisible until the SDK loads. LCP = SDK load time, up to the timeout
FCP	Same
INP	The SDK is 30–60 KB of blocking main‑thread work
CLS	If content swaps after reveal, you get a shift

A 4‑second anti‑flicker timeout means a 4‑second worst‑case LCP for every user, including the control group who see no change at all. Teams run this for years without connecting it to their LCP problem.

And there's a compounding irony: the experiment platform makes every experiment look worse, because both variants are slowed by the platform. You get a systematically biased read on every test you run.

Edge assignment: the fix¶

Decide the variant before the HTML is generated. Nothing to hide, nothing to swap, nothing to flicker.

// middleware.ts (or proxy.ts on Next.js 16)
import { NextRequest, NextResponse } from 'next/server';

type Experiment = { key: string; variants: string[]; weights: number[]; paths: RegExp };

const EXPERIMENTS: Experiment[] = [
  { key: 'pdp_gallery_v2', variants: ['control', 'treatment'], weights: [0.5, 0.5], paths: /^\/p\// },
  { key: 'plp_density',    variants: ['control', 'dense'],     weights: [0.9, 0.1], paths: /^\/c\// },
];

export function middleware(req: NextRequest) {
  const res = NextResponse.next();

  // Stable anonymous ID, set once
  let anonId = req.cookies.get('anon_id')?.value;
  if (!anonId) {
    anonId = crypto.randomUUID();
    res.cookies.set('anon_id', anonId, {
      maxAge: 60 * 60 * 24 * 365, sameSite: 'lax', path: '/', httpOnly: true,
    });
  }

  const assignments: string[] = [];
  for (const exp of EXPERIMENTS) {
    if (!exp.paths.test(req.nextUrl.pathname)) continue;
    // Deterministic hash — same user always gets the same variant, no storage needed
    const variant = pickVariant(`${exp.key}:${anonId}`, exp.variants, exp.weights);
    res.headers.set(`x-exp-${exp.key}`, variant);
    assignments.push(`${exp.key}=${variant}`);
  }

  // One low-cardinality header for the cache key + analytics
  if (assignments.length) res.headers.set('x-experiments', assignments.join(','));
  return res;
}

function pickVariant(seed: string, variants: string[], weights: number[]): string {
  // FNV-1a → [0, 1)
  let h = 2166136261;
  for (let i = 0; i < seed.length; i++) {
    h ^= seed.charCodeAt(i);
    h = Math.imul(h, 16777619);
  }
  const r = (h >>> 0) / 4294967296;
  let acc = 0;
  for (let i = 0; i < variants.length; i++) {
    acc += weights[i];
    if (r < acc) return variants[i];
  }
  return variants[0];
}

export const config = { matcher: ['/p/:path*', '/c/:path*'] };

// app/p/[slug]/page.tsx — read the assignment server-side
import { headers } from 'next/headers';

export default async function ProductPage({ params }) {
  const variant = (await headers()).get('x-exp-pdp_gallery_v2') ?? 'control';
  const product = await getProduct((await params).slug);

  // The correct variant is in the initial HTML. No flicker, no CLS, no SDK.
  return variant === 'treatment'
    ? <ProductGalleryV2 product={product} />
    : <ProductGallery product={product} />;
}

Cost: ~1 ms of middleware, zero client JS, zero flicker.

The cache‑fragmentation trade‑off¶

Server‑side variants multiply your cache entries.

1 experiment × 2 variants                        = 2× cache entries
3 experiments × 2 variants                       = 8× cache entries
3 experiments × 2 variants × 5 markets           = 40× cache entries  ← too far

Rules that keep this manageable:

Cap concurrent experiments per page type at 2–3. This is also good experiment hygiene — interaction effects between many simultaneous tests make results unreliable anyway.
Only include experiments in the cache key when they change the HTML. An experiment affecting only a client‑side behavior doesn't need a separate cached page.
Use a compact, normalized header (x-experiments: a=1,b=0) rather than many headers.
Sunset experiments aggressively. A finished experiment left in the code is permanent cache fragmentation. Enforce an expiry date.

// Only experiments that alter server-rendered output participate in the cache key
const CACHE_AFFECTING = new Set(['pdp_gallery_v2', 'plp_density']);

const cacheKeyExperiments = assignments
  .filter((a) => CACHE_AFFECTING.has(a.split('=')[0]))
  .sort()                     // stable ordering
  .join(',');
res.headers.set('x-cache-variant', cacheKeyExperiments);
// CDN: add x-cache-variant to the cache key, and nothing else

Client‑side flags, done acceptably¶

Sometimes the variant genuinely can't be server‑decided (it depends on client state). Minimize the damage:

// ❌ Hides content until the flag resolves
{flagsLoaded ? (variant === 'b' ? <NewHero /> : <OldHero />) : null}

// ✅ Render the control immediately; enhance if the treatment applies.
//    Both variants must occupy the same space so there's no CLS.
<div className="min-h-[420px]">
  {variant === 'b' ? <NewHero /> : <OldHero />}
</div>

Rules: 1. Never hide the page waiting for flags. 2. Default to control, synchronously, with no loading state. 3. Both variants must have identical dimensions, or you've built a CLS bug. 4. Bootstrap flags from a server‑rendered payload, not a client fetch:

// app/layout.tsx — flags resolved server-side, inlined, no round trip
export default async function RootLayout({ children }) {
  const flags = await resolveFlags();      // from cookies/headers, cached
  return (
    <html>
      <body>
        <FlagProvider initial={flags}>{children}</FlagProvider>
      </body>
    </html>
  );
}

Dead code from flags¶

Flags accumulate. Six months later you're shipping both branches of twelve resolved experiments to every user.

// ❌ Both components ship in the bundle forever
import { GalleryV1 } from './gallery-v1';
import { GalleryV2 } from './gallery-v2';
const Gallery = flag ? GalleryV2 : GalleryV1;

// ✅ Only the used variant is downloaded
const Gallery = flag
  ? dynamic(() => import('./gallery-v2'))
  : dynamic(() => import('./gallery-v1'));

For server‑decided variants, the Server Component approach is better still — the unused branch never reaches the client at all.

Flag hygiene process:

// lib/flags/registry.ts — every flag declares an expiry
export const FLAGS = {
  pdp_gallery_v2: {
    description: 'New PDP gallery with pinch zoom',
    owner: '@pdp-squad',
    created: '2026-02-14',
    expires: '2026-05-14',        // enforced by CI
    type: 'experiment',
  },
  enable_new_checkout: {
    description: 'Rewritten checkout flow',
    owner: '@checkout-squad',
    created: '2026-01-08',
    expires: '2026-04-08',
    type: 'release',
  },
} as const;

// scripts/check-expired-flags.mjs — fails CI on expired flags
import { FLAGS } from '../lib/flags/registry.js';

const expired = Object.entries(FLAGS)
  .filter(([, f]) => new Date(f.expires) < new Date())
  .map(([key, f]) => `${key} (owner ${f.owner}, expired ${f.expires})`);

if (expired.length) {
  console.error('Expired feature flags must be removed:\n  ' + expired.join('\n  '));
  process.exit(1);
}

This is the control that keeps the codebase and the cache from degrading. Without it, flag debt is permanent.

Measuring performance changes as experiments¶

Every performance change should be an experiment. Here's the discipline.

Pre‑registration¶

## Experiment: PDP static shell (PPR)

**Hypothesis.** Reducing PDP p75 LCP from 4.6s to ~2.2s increases conversion.

**Primary metric.** Conversion rate (session → order).
**MDE.** 1% relative. **Power.** 80%. **α.** 0.05.
**Required sample.** ~1.8M sessions per arm ≈ 14 days at current traffic.

**Guardrails (any breach → stop):**
- Revenue per session: no decrease
- JS error rate: no increase > 10% relative
- Add-to-cart rate: no decrease
- Price-mismatch reports: zero
- p99 TTFB: no increase

**Health checks (verify the change actually happened):**
- p75 LCP in the treatment arm must be < 2.6s. If not, the deploy didn't work —
  stop and fix before drawing any conclusion.

**Analysis date.** 2026-03-14. No peeking before then except for guardrails.

The health check line is the one people skip and the one that saves you. A surprising fraction of "speed doesn't affect conversion" conclusions come from experiments where the treatment arm wasn't actually faster.

Implementation¶

// The variant is decided at the edge; both paths are fully server-rendered
export default async function ProductPage({ params }) {
  const variant = (await headers()).get('x-exp-pdp_ppr') ?? 'control';
  const { slug } = await params;
  return variant === 'treatment'
    ? <ProductPagePPR slug={slug} />
    : <ProductPageLegacy slug={slug} />;
}

// RUM must record the variant, or you can't split the metrics
reportMetric({
  name: metric.name,
  value: metric.value,
  dimensions: {
    page_type: 'pdp',
    device_class: deviceClass,
    experiments: document.documentElement.dataset.experiments,   // set server-side
    release_sha: process.env.NEXT_PUBLIC_RELEASE_SHA,
  },
});

Analysis¶

-- Performance delta, per arm
SELECT variant,
       COUNT(*) AS samples,
       APPROX_QUANTILES(value, 100)[OFFSET(50)] AS p50,
       APPROX_QUANTILES(value, 100)[OFFSET(75)] AS p75,
       APPROX_QUANTILES(value, 100)[OFFSET(95)] AS p95
FROM vitals
WHERE name = 'LCP' AND page_type = 'pdp' AND device_class = 'mobile'
  AND experiment = 'pdp_ppr'
GROUP BY variant;

-- Business delta, same population
SELECT variant,
       COUNT(DISTINCT session_id) AS sessions,
       COUNT(DISTINCT order_id) / COUNT(DISTINCT session_id) AS conversion,
       SUM(revenue) / COUNT(DISTINCT session_id) AS revenue_per_session
FROM sessions_with_experiments
WHERE experiment = 'pdp_ppr'
GROUP BY variant;

Read them together. If LCP improved 2.4 s and conversion is flat, that's a real and useful finding — it tells you your bottleneck is elsewhere. Report it honestly. A team that only reports wins loses the ability to make decisions from data.

Statistical honesty¶

Practice	Why
Pre‑register the primary metric	Otherwise you'll find something significant
Don't peek (except guardrails)	Repeated testing inflates false positives
Run for whole weeks	Retail has strong day‑of‑week and payday effects
Sticky assignment per user	Or returning users see both arms
Check sample‑ratio mismatch	A 50/50 split arriving as 53/47 means the assignment is broken
Report null results	The whole point of measuring
One change per experiment	Otherwise you can't attribute

-- Sample ratio mismatch check — run this before analyzing anything else
SELECT variant, COUNT(*) AS n,
       COUNT(*) / SUM(COUNT(*)) OVER () AS share
FROM sessions_with_experiments WHERE experiment = 'pdp_ppr' GROUP BY variant;
-- Expect 0.50/0.50. Anything beyond ~0.51/0.49 at large n means the
-- assignment or the logging is broken — fix it before trusting the result.

Aurora's experimentation change¶

	Before	After
Assignment	Client SDK with a 4 s anti‑flicker timeout	Edge, deterministic hash
Client JS	34 KB	0
LCP impact of the platform	+1.2 s worst case	0
CLS from variant swaps	0.06	0
Cache entries per page	1 (uncacheable — SDK read cookies)	2–4
Time to launch an experiment	10 min (marketing self‑serve)	1 deploy

That last row is the real cost, and it's worth naming. Growth teams value self‑serve. The compromise Aurora reached: edge assignment for anything that changes server‑rendered HTML; the vendor SDK retained for copy‑only and styling‑only tests, loaded with lazyOnload and with the anti‑flicker snippet removed (copy changes below the fold don't need it).

That kept 80% of the experiment velocity and removed 100% of the LCP damage.

Common mistakes¶

Mistake	Cost
Anti‑flicker snippets	LCP = SDK load time, for everyone
Client‑side variant swapping	CLS + flicker
Not recording the variant in RUM	Can't analyze performance by arm
No health check that the treatment is actually faster	False "speed doesn't matter" conclusions
Peeking at results	Inflated false positives
Running 8 experiments on one page	Interaction effects; cache fragmentation
Flags with no expiry	Permanent dead code and cache splits
Shipping both branches statically	Bundle carries every resolved experiment
Not checking sample ratio	Analyzing a broken experiment

Lab 8.5 — Fix your experimentation¶

Measure the platform's cost: WebPageTest with the A/B vendor's domain blocked. The LCP/TBT delta is what your experiment platform costs every user.
Find the anti‑flicker snippet. If present, this is your top LCP item — treat it as such.
Implement edge assignment for one experiment. Verify no flicker and no CLS.
Add the variant dimension to RUM and confirm you can query p75 per arm.
Audit your flags: how many exist, how many are past their decision date, how many ship both branches? Build the registry and the CI expiry check.
Pre‑register your next performance change as an experiment, including the health check that the treatment arm actually got faster.

Checklist¶

Next: 8.6 Infrastructure & multi‑region