1.4 — Measuring: lab vs field¶

Module 1 · Lesson 4 · 🟢 Foundational · ~35 min

What you'll learn¶

Which tool answers which question, and where each one lies
How to read a Chrome DevTools performance trace like an engineer, not a tourist
How to run a controlled before/after comparison you can defend
The measurement mistakes that produce confidently wrong conclusions

The tool map¶

Tool	Type	Answers	Lies about
RUM (your own)	Field	What real users experience, segmented any way you like	Nothing — but only if you instrument correctly
CrUX / PageSpeed Insights field data	Field	Chrome users' p75, public, per‑origin and per‑URL	Small‑traffic pages (no data), non‑Chrome users, 28‑day lag
Lighthouse	Lab	A reproducible score + specific diagnostics	INP, real device mix, real network, cache states
DevTools Performance panel	Lab	Exactly what the main thread did, millisecond by millisecond	Your users' devices (unless you throttle)
WebPageTest	Lab	Real devices, real networks, filmstrips, request blocking	Cost/time per run; still not your traffic
Next.js build output / bundle analyzer	Static	Bytes per route, chunk composition	Runtime cost, execution time
Server APM / traces	Field	Where server time goes	Client‑side anything

The workflow that works:

Field (RUM/CrUX)  →  "PDP p75 LCP is 4.6s on mobile; 44% is resource load time"
        ↓
Lab (DevTools/WPT) →  "because the hero is a 340KB JPEG at 2200px on a 390px viewport"
        ↓
Fix + lab verify   →  "prototype shows 1.9s"
        ↓
Field verify (A/B) →  "shipped to 50%: p75 4.6s → 2.3s, conversion +1.4%"

Skipping the first step means optimizing something that isn't broken. Skipping the last means you never know if it worked.

Field measurement¶

CrUX — free, public, coarse¶

The Chrome User Experience Report aggregates real Chrome users' vitals. Available through PageSpeed Insights, the CrUX API, BigQuery, and Search Console.

curl -s "https://chromeuxreport.googleapis.com/v1/records:queryRecord?key=$CRUX_API_KEY" \
  -H 'Content-Type: application/json' \
  -d '{
    "origin": "https://www.auroramarket.com",
    "formFactor": "PHONE"
  }' | jq '.record.metrics | to_entries[] | {metric: .key, p75: .value.percentiles.p75}'

Use CrUX for: competitive benchmarking, executive reporting, and a sanity check that your RUM isn't miscalibrated. Don't use it for day‑to‑day work — it's a 28‑day rolling window, so a fix takes a month to fully show up, and it only covers Chrome users who opt into reporting.

You can also query by URL pattern group rather than origin, which is more useful for a commerce site where PDP and checkout behave completely differently.

RUM — your own, precise¶

This is your primary instrument. Full implementation in 9.1; the short version:

// app/components/vitals.tsx
'use client';
import { useReportWebVitals } from 'next/web-vitals';

export function WebVitals() {
  useReportWebVitals((metric) => {
    navigator.sendBeacon('/api/vitals', JSON.stringify({
      name: metric.name,
      value: metric.value,
      rating: metric.rating,
      id: metric.id,
      path: window.location.pathname,
    }));
  });
  return null;
}

That's the minimum. The version you actually want uses web-vitals/attribution and adds dimensions — page type, device class, connection type, logged‑in state, A/B variant, release SHA, and cache status. Dimensions are what turn a number into an action.

The release SHA dimension is the one people forget. With it, "LCP regressed on Tuesday" becomes "LCP regressed in deploy a3f9c21", and triage takes ten minutes instead of two days.

Lab measurement¶

Lighthouse: useful, widely misused¶

Run it from the CLI for reproducibility, never from the DevTools panel for anything you'll report:

npx lighthouse https://www.auroramarket.com/p/wool-overshirt-navy \
  --preset=desktop \
  --output=json --output=html --output-path=./lh-desktop \
  --chrome-flags="--headless=new"

# Mobile (the default preset) with explicit throttling
npx lighthouse https://www.auroramarket.com/p/wool-overshirt-navy \
  --form-factor=mobile \
  --throttling-method=simulate \
  --output=json --output-path=./lh-mobile.json

What Lighthouse is good for: - The Diagnostics and Opportunities sections — specific, actionable, usually correct - "Avoid enormous network payloads", "Properly size images", "Reduce unused JavaScript" - Third‑party summary and main‑thread breakdown by script - Reproducible CI gating (with the caveats in 9.2)

What Lighthouse is bad for: - INP (it doesn't measure it — TBT is a proxy) - Predicting field results (simulated throttling ≠ real networks) - A single number to manage a team by - Run‑to‑run stability: variance is ±5–10 points, so run 5 times and take the median

# Median of 5 — the only defensible way to compare Lighthouse runs
for i in $(seq 1 5); do
  npx lighthouse "$URL" --quiet --output=json --output-path="./lh-$i.json" \
    --chrome-flags="--headless=new"
done
jq -s 'map(.audits["largest-contentful-paint"].numericValue) | sort | .[2]' lh-*.json

DevTools Performance panel: the real instrument¶

This is where you actually find things. The workflow:

Set throttling first. CPU 4× (or 6× for a realistic mid‑tier Android), Network "Slow 4G". Do this before recording, every time.
Record a page load: click the reload‑and‑record button. For interactions, start recording, perform one interaction, stop.
Read it in this order:

Track	What to look for
Timings	FCP, LCP, DCL markers. Where does LCP land relative to script activity?
Network	The critical chain. Is the LCP resource requested early? Are there dependent chains?
Main	Long tasks (red‑cornered). Their width is your INP problem.
Interactions	Present in recent Chrome — shows each interaction's duration and sub‑parts
Bottom‑Up (grouped by URL)	Which script owns the main‑thread time. This is the money view for third parties
Call Tree	Which function inside that script

Key trick — group Bottom‑Up by "Product" or by URL. In three seconds you learn that 41% of scripting is the tag manager, which reframes the entire project.
For React specifically, use the React DevTools Profiler alongside it (see 5.1). The browser trace tells you that a 400 ms task happened; React's profiler tells you which component tree caused it.

WebPageTest: for the hard questions¶

Worth learning for three capabilities you can't get elsewhere:

1. Real devices on real networks. Run on an actual mid‑tier Android on a real 4G connection in the geography your users are in.

2. Request blocking — the fastest way to quantify a third party. Run the page normally, then run it again blocking the tag manager domain. The delta is the cost, and it's a number nobody can argue with.

# WebPageTest script: measure the cost of the tag stack
blockDomains  tagmanager.example.com analytics.example.com chat-vendor.example.com
navigate      https://www.auroramarket.com/p/wool-overshirt-navy

3. Filmstrips and visual comparison. Two runs side by side, frame by frame, is the most persuasive artifact you can put in a slide deck. Executives don't read flamegraphs; they understand a filmstrip where one row shows a product at 1.8 s and the other shows white at 4.0 s.

Running a defensible before/after¶

Most "we made it 40% faster" claims fall apart under questioning. Here's how to make one that doesn't.

In the lab:

#!/usr/bin/env bash
# scripts/perf-compare.sh — median-of-N comparison between two URLs/branches
set -euo pipefail
BASELINE_URL="$1"; CANDIDATE_URL="$2"; RUNS="${3:-7}"

measure() {
  local url="$1" label="$2"
  for i in $(seq 1 "$RUNS"); do
    npx lighthouse "$url" --quiet --only-categories=performance \
      --output=json --output-path="/tmp/$label-$i.json" \
      --chrome-flags="--headless=new" >/dev/null
  done
  jq -s '
    map({
      lcp: .audits["largest-contentful-paint"].numericValue,
      tbt: .audits["total-blocking-time"].numericValue,
      cls: .audits["cumulative-layout-shift"].numericValue,
      bytes: .audits["total-byte-weight"].numericValue
    })
    | { lcp: (map(.lcp) | sort | .[length/2|floor]),
        tbt: (map(.tbt) | sort | .[length/2|floor]),
        cls: (map(.cls) | sort | .[length/2|floor]),
        bytes: (map(.bytes) | sort | .[length/2|floor]) }
  ' /tmp/"$label"-*.json
}

echo "baseline:";  measure "$BASELINE_URL"  baseline
echo "candidate:"; measure "$CANDIDATE_URL" candidate

Controls that matter: same machine, same time window (network conditions drift), no other apps running, headless, cache disabled, at least 5–7 runs, compare medians not means.

In the field: an A/B holdback with the flag recorded as a RUM dimension, so you can query p75 per variant directly:

SELECT variant,
       COUNT(*) AS samples,
       APPROX_QUANTILES(value, 100)[OFFSET(75)] AS p75_lcp
FROM vitals
WHERE name = 'LCP' AND page_type = 'pdp' AND device_class = 'mobile'
  AND ts >= TIMESTAMP_SUB(CURRENT_TIMESTAMP(), INTERVAL 14 DAY)
GROUP BY variant;

If the variant's p75 didn't move in the field, the change didn't work — regardless of what the lab said. This happens more often than you'd expect, usually because the fix only helped cold‑cache first‑time visitors who are a minority of your traffic, or because a third party grew to fill the space you freed.

Common mistakes¶

Mistake	Consequence
Comparing single Lighthouse runs	±5–10 points of noise; you'll "prove" random changes
Testing on your dev machine unthrottled	You cannot perceive the problem your users have
Measuring localhost	No network latency, no CDN, dev builds are 3–10× slower and not representative
Testing with a warm cache	Hides the first‑visit experience, which is most of your SEO traffic
Ignoring the p75/p95 tail	The tail is where abandonment happens
Not segmenting by page type	PDP and checkout have opposite bottlenecks; the average hides both
Measuring only Chrome	Safari/iOS is a large share of commerce traffic and doesn't report to CrUX
Optimizing without a before number	You can't prove the win, so the work gets defunded

Never benchmark a Next.js dev build. next dev does no minification, includes HMR, and compiles routes on demand. Always measure next build && next start, or a preview deployment.

Lab 1.4 — Set up your measurement stack¶

Field: ship the RUM reporter (attribution build) with at minimum these dimensions: page_type, device_class, connection, is_returning, release_sha, ab_variant.
Lab: create scripts/perf-compare.sh from the snippet above and run it against production for your top three page types. Commit the baseline JSON to the repo with a date.
DevTools: record one throttled trace per page type. In each, write down:
the three longest tasks and their owning script,
the LCP element and when it was requested vs painted,
total scripting time before LCP.
WebPageTest: run your PDP twice — once normal, once with all third‑party domains blocked. Record the LCP and TBT delta. Put the filmstrip comparison in a doc.
CrUX: pull your origin's p75 and one competitor's. Note the gap.

You now have everything you need to prioritize. Everything after this is execution.

Checklist¶

RUM live with attribution and the six core dimensions
Baseline lab numbers committed to the repo, dated, reproducible via a script
CPU + network throttling is your default DevTools state
You know the cost of your third‑party stack as a specific ms number
Every perf PR includes a before/after with medians, not single runs
Nobody compares dev‑build numbers to production

Next: 1.5 Performance budgets