Testing an interface that changes every time

A generated flight-results page can lose a price when its layout changes. I wanted a check that would catch that.

I built uivet to check interfaces produced by a generator. The same prompt can produce different layouts, so I wanted to check whether the result still contains the information it was given and whether the rendered page has detectable problems.

I wanted the check to survive a layout change. A pixel comparison would tell me that the page changed, but I also needed to know whether it still showed the supplied data.

Check the output, several times

A uivet scenario contains a prompt, input data, and a number of runs. Each generated HTML document is rendered in Chromium using Playwright. The harness checks data fidelity, accessibility, layout heuristics, and console errors, then aggregates the results.

The input data gives the fidelity check something specific to look for. If the flight data includes a fare of $1,248, the rendered page should preserve it. Looking polished does not compensate for omitting it.

You can follow the implementation alongside this piece: the checks, the runner, and the offline configuration. These links point to the revision used below.

Sampling several outputs helps reveal variation, but three successful samples are not a guarantee about the fourth. The report is evidence about the pages that actually ran.

Run it without a model

The offline demo replays recorded HTML and switches off the model judge. It needs no API key and makes no model calls. From a fresh clone, with Bun installed:

git clone https://github.com/MaryanPrydatko/uivet.git
cd uivet
git checkout 66c67076e841bbb78128804b06b936ab3d2b9605
bun install --frozen-lockfile
bunx playwright install chromium
bun run demo:offline --out results/offline

Open results/offline/report.html to inspect the results. The browser installation is a download; “offline” refers to replaying the fixtures without generating new interfaces or calling a model.

At the revision checked for this article, the three scenarios each ran three times and the demo exited successfully. The older recording in the README shows a missing-price failure. The current fixture set does not reproduce that failure by itself, so here is an explicit way to test it.

Remove a price and expect failure

Save this as check-missing-price.ts in that fresh checkout, then run bun check-missing-price.ts. It removes a price from the third flight fixture, runs the harness, and restores the original fixture in a finally block. It checks both the exit code and the reported missing value, so an unrelated startup error cannot count as catching the bug.

import assert from 'node:assert/strict';

const path = 'examples/fixtures/flight-results/2.html';
const original = await Bun.file(path).text();
const output = `results/missing-price-${crypto.randomUUID()}`;
assert.ok(original.includes('$1,248'));

try {
  await Bun.write(path, original.replaceAll('$1,248', ''));
  const run = Bun.spawn([
    'bun', 'run', 'demo:offline', '--out', output,
  ], { stdout: 'inherit', stderr: 'inherit' });
  assert.equal(await run.exited, 1);

  const report = await Bun.file(`${output}/results.json`).json();
  const flight = report.scenarios.find(s => s.id === 'flight-results');
  assert.ok(flight.runs[2].fidelity.missing.includes('$1,248'));
  console.log(`The missing price failed the gate. Report: ${output}/report.html`);
} finally {
  await Bun.write(path, original);
}

The nested demo command should exit with code 1. The checking script itself should succeed and print “The missing price failed the gate.” Each run uses a new directory under results/, printed by the script, so an old report cannot accidentally satisfy the assertion. Run this experiment on its own so another test is not reading the fixture while it is temporarily changed.

The script establishes a specific failure: that fixture lost a price and the fidelity gate caught it. The next experiment shows what the same gate can miss.

Swap the prices and watch the check pass

The fidelity check walks the scenario data, collects string and numeric leaves, normalises the rendered text, and looks for each value. That makes it independent of a generated component’s structure. I do not have to predict which classes or elements a model will choose.

The tradeoff is that it measures presence, not relationships. Here is an intentionally wrong result that still gets full fidelity. Save it as check-fidelity-limit.ts in the same pinned checkout and run it with bun check-fidelity-limit.ts:

import assert from 'node:assert/strict';
import { computeFidelity } from './src/checks.ts';

const data = {
  flights: [
    { airline: 'SWISS', price: '$1,248' },
    { airline: 'United', price: '$986' },
  ],
};
const swappedPrices = 'SWISS $986\nUnited $1,248';
const result = computeFidelity(data, swappedPrices);
assert.equal(result.rate, 1);
assert.deepEqual(result.missing, []);
console.log('Every value is present, but the prices belong to the wrong flights.');

To catch this, the check would have to identify each flight row and verify its fare within that row. That creates another requirement: a stable way to locate a flight across different generated layouts. I would add that to the scenario’s contract before claiming to verify the relationship.

The aggregate needs care too. uivet averages the fidelity rates across runs. In the missing-price experiment, two runs find all 25 expected values and the third finds 24. The mean is about 98.67%, displayed as 99%. That looks close to perfect, but the requirement was to preserve every value. The default threshold of 1 rejects it. Lowering the threshold would change what the green result promises.

I also keep the individual runs in the report. An average tells me less than seeing which generation lost which value. A model’s occasional failure can matter more to a user than its average score suggests.

Decide which observations should block a run

The report and the exit code answer different questions. At the pinned revision, the policy is:

My successful offline run still reported serious accessibility issues. “PASS” meant it met those configured gates; it did not mean the page was accessible or ready to ship.

A saved baseline adds another comparison. The runner detects a lower fidelity rate and newly appearing accessibility rule IDs, even when a run remains above its absolute thresholds. Judge-score regression is advisory unless judge enforcement is enabled. Updating the baseline accepts a new comparison point, so I would review that change along with the implementation.

The prototype renders at 1280 × 800 and does not cover mobile layouts. Its text matching is English-oriented and sensitive to formatting. The swapped-price example shows why a perfect presence score still needs a more specific check before it can approve an interface.

For a new scenario, I would start with an output I expect to pass, introduce a known failure, and check which part of the report and exit policy catches it. Then I would try a wrong output that preserves the obvious values, like the swapped prices above. I can then decide whether to strengthen the check or add another one for the behaviour it misses.