QA and QoE

When to use it: deciding what "tested" means before a release, or arguing
about whether a slow-but-working build should ship. Companion to
QA and QoE Testing with Playwright MCP, which is the how.

There is a number in video engineering that has embarrassed the industry for
twenty years. PSNR — peak signal-to-noise ratio — measures how much a compressed
frame differs, mathematically, from the original. It is objective, cheap, and was
for decades the default way to compare codecs.

Netflix published the chart that killed it. At a PSNR of roughly 35 dB, the
human-rated quality of their test clips ranged from 10 to 100 on a 100-point
scale — from "impairments are annoying" to "impairments are imperceptible". The
same measurement. Opposite experiences. Their conclusion was blunt: "PSNR does
not consistently reflect human perception," even though it remained "the de
facto standard for codec comparisons."

That gap is the whole subject. PSNR is a QA metric wearing a QoE costume. It
answers is the output correct? with real rigour, and it does not answer is
this any good to watch? at all.

Two different questions

QA asks whether the thing works. It is binary, per-build, and it has a right
answer. The form submits or it doesn't. The endpoint returns 200 or 500. The
link is dead or alive. You can gate a merge on it, because "broken" is not a
matter of opinion.

QoE asks what it is like to be the person using it. It is continuous,
distributional, and it has no pass mark handed down from the code. Two seconds
to first frame is not a bug. Six seconds is not a bug either — nothing threw,
nothing returned an error, every test is green. It is just worse, and past some
threshold people leave.

Most engineering organisations have a mature answer to the first question and
approximately nothing for the second. The test suite is the QA function, it runs
on every commit, and a red build blocks the merge. QoE, if it is measured at all,
is a dashboard someone looks at after a complaint.

The asymmetry is not because QoE matters less. It is because QA is easy to
automate and QoE is easy to argue about.

What Netflix actually measures

Netflix runs A/B tests on the streaming path itself, and the metric set they
report on each playback session is worth reading closely:

play delay; the rates of rebuffers (playback interruptions when the video
buffer empties), playback errors, and user-initiated aborts; the average
bitrate throughout playback; and Video Multimethod Assessment Fusion, a
measure of perceptual video quality developed here at Netflix.

Six metrics. Exactly one of them — playback errors — is a QA metric in the
ordinary sense. The other five are all degrees. Play delay is not pass/fail.
Rebuffer rate is a ratio: how much of the play time was spent staring at a
spinner. User-initiated aborts are people voting with the back button. Average
bitrate is how sharp the picture was. VMAF is a perceptual quality score.

A company whose entire product is "video comes out of the internet" decided that
five-sixths of what to watch is not whether it worked.

Note also what rebuffer rate implies structurally. It is a fraction of play
time, not a count. That framing is deliberate: it makes the metric comparable
between a 22-minute sitcom and a three-hour film, which a raw count of stalls
would not be. Choosing the denominator is most of the work in defining a QoE
metric, and it is the step teams usually skip.

The part that costs money

The hard evidence that QoE drives behaviour does not come from Netflix — it
comes from an Akamai study that gets misattributed to them constantly, so it is
worth citing properly. Krishnan and Sitaraman analysed 6.7 million viewers,
23 million videos, 216 million minutes of streaming across ten days, and used
quasi-experimental designs to argue causality rather than correlation. Three
findings:

Viewers begin abandoning a video once startup exceeds about two seconds,
and each additional second of delay increases abandonment by ~5.8%.
A viewer who experiences rebuffering equal to 1% of the video's duration
watches 5% less of it than an otherwise-similar viewer with no rebuffering.
A viewer who hits a failure is 2.32% less likely to return to the same site
within a week.

Read the third one again. The cost of a bad experience is not confined to the
session in which it happened — it shows up a week later as someone who didn't
come back. That is the number that makes QoE a business metric rather than an
engineering nicety, and it is the number no test suite will ever produce for
you, because nothing was broken.

Why averages lie

The subtlest thing in Netflix's experimentation write-up is methodological. They
observe that "treatment effects in these streaming experiments tend to be
heterogeneous with respect to network conditions and other factors," and
conclude that "changes in the mean, the median, or other summary statistics are
not generally sufficient." So they compare whole distributions — quantile
functions plotted in the metric's own units, with bootstrap confidence intervals
and Bonferroni corrections across the quantiles.

This matters far beyond video. A change that improves the mean can be a
catastrophe: shave 100 ms off startup for the 80% of users on fibre while adding
two seconds for the 20% on a bad mobile connection, and your average improves
while you actively make things worse for the people already suffering most. QoE
regressions concentrate. They land on old devices, slow networks, and distant
regions — populations that are, by construction, underrepresented in the
office where the change was written and tested.

This is why Core Web Vitals are specified at the 75th percentile rather than
the mean. The interesting question was never "how is it for a typical user" but
"how bad is it for the unlucky quarter."

If you take one operational habit from this essay: report QoE as a percentile,
never as an average, and segment by device and connection before you believe
any improvement.

Optimising the metric you actually care about

The Netflix story has a satisfying ending that most quality efforts don't,
because they did the expensive thing: having established that PSNR didn't
predict experience, they built a metric that does.

VMAF is a machine-learned fusion of several elementary quality signals, trained
against human opinion scores — real observers rating impairments on a
standardised scale, converted into differential mean opinion scores. It exists
because the honest measurement was the one that correlated with people, and no
formula they had did that.

Then they optimised against it. Per-title encoding (2015) abandoned the fixed
bitrate ladder on the grounds that every title has its own relationship between
bitrate and perceived quality — reportedly 15–20% bandwidth saved at equal VMAF.
Per-shot encoding via the Dynamic Optimizer went further, choosing resolution
and quantiser per shot to maximise predicted VMAF under a bitrate budget, for
roughly another 10–15%. The cost was brutal: per-shot multiplies the number of
encodes by a hundred or more.

That trade is the point. They spent enormous compute to buy bitrate reduction at
constant perceived quality — an optimisation that is invisible to QA. Every
frame decoded correctly before and after. Nothing was broken; something was
better.

What this means if you are not Netflix

You almost certainly do not need VMAF. The transferable parts are cheap:

Pick your six. Netflix's set maps onto ordinary web software with very little
imagination. Play delay is LCP. Rebuffer rate is time-to-interactive jank and
long tasks. Playback errors are your JS exceptions and 5xxs. User-initiated
aborts are bounces and rage-clicks. Bitrate is the quality you degraded to under
load. VMAF is whatever proxy you have for "did this look right" — visual
regression, layout shift.

Give each one a threshold and a denominator before you start collecting, or
you will end up with a dashboard nobody can act on. "LCP p75 under 2.5s on
mobile" is actionable. "The site feels fast" is not.

Measure it in CI, not in a postmortem. A QoE number produced only after a
complaint is a QoE number that has already cost you something. Driving a real
browser against a real build on every significant change is the practical
version of this — see QA and QoE Testing with Playwright MCP for the
mechanics.

Report the two separately. This is the habit that makes the rest work. A run
that finds no QA defects and an LCP of six seconds is not a pass, and any report
format that lets it be recorded as one will eventually record it as one.

The uncomfortable part

QoE work is harder to justify than QA work, and the reason is structural rather
than political. A failing test names its own remedy: something is broken, fix it,
the build goes green, done. A QoE regression names nothing. It is a number that
got worse, on a distribution, for a subset of users, with no obvious threshold at
which it becomes unacceptable — and the same ambiguity that makes it easy to
deprioritise is exactly what makes it accumulate.

Nobody ever shipped the change that made the app slow. Everybody shipped the
forty changes that each made it 3% slower, every one of them passing QA.

That is what QoE measurement is for: making the slow accumulation visible while
each increment is still individually cheap to reverse. Not because latency is
more important than correctness — an app that loses your data instantly is worse
than one that saves it slowly — but because correctness already has a machine
watching it, and experience usually doesn't.

QA tells you the build is correct. QoE tells you whether anyone will stay.