Engineering9 September 2026

Testing an interface that changes every time

“Make it look better” is easy to ask. It is harder to notice that the new version has quietly lost a price.

I built uivet to check interfaces produced by a generator. The same prompt can produce different layouts, so I wanted to check whether the result still contains the information it was given and whether the rendered page has detectable problems.

A screenshot is useful to inspect. A pixel difference alone does not tell me whether a changed layout is an improvement or whether it broke a requirement.

Check the output, several times

A uivet scenario contains a prompt, input data, and a number of runs. Each generated HTML document is rendered in Chromium using Playwright. The harness checks data fidelity, accessibility, layout heuristics, and console errors, then aggregates the results.

The input data gives the fidelity check something specific to look for. If the flight data includes a fare of $1,248, the rendered page should preserve it. Looking polished does not compensate for omitting it.

You can follow the implementation alongside this piece: the checks, the runner, and the offline configuration. These links point to the revision used below.

Sampling several outputs helps reveal variation, but three successful samples are not a guarantee about the fourth. The report is evidence about the pages that actually ran.

Run it without a model

The offline demo replays recorded HTML and switches off the model judge. It needs no API key and makes no model calls. From a fresh clone, with Bun installed:

git clone https://github.com/MaryanPrydatko/uivet.git
cd uivet
git checkout 66c67076e841bbb78128804b06b936ab3d2b9605
bun install --frozen-lockfile
bunx playwright install chromium
bun run demo:offline --out results/offline

Open results/offline/report.html to inspect the results. The browser installation is a download; “offline” refers to replaying the fixtures without generating new interfaces or calling a model.

At the revision checked for this article, the three scenarios each ran three times and the demo exited successfully. The older recording in the README shows a missing-price failure. The current fixture set does not reproduce that failure by itself, so here is an explicit way to test it.

Remove a price and expect failure

Save this as check-missing-price.ts in that fresh checkout, then run bun check-missing-price.ts. It removes a price from the third flight fixture, runs the harness, and restores the original fixture in a finally block. It checks both the exit code and the reported missing value, so an unrelated startup error cannot count as catching the bug.

import assert from 'node:assert/strict';

const path = 'examples/fixtures/flight-results/2.html';
const original = await Bun.file(path).text();
const output = `results/missing-price-${crypto.randomUUID()}`;
assert.ok(original.includes('$1,248'));

try {
  await Bun.write(path, original.replaceAll('$1,248', ''));
  const run = Bun.spawn([
    'bun', 'run', 'demo:offline', '--out', output,
  ], { stdout: 'inherit', stderr: 'inherit' });
  assert.equal(await run.exited, 1);

  const report = await Bun.file(`${output}/results.json`).json();
  const flight = report.scenarios.find(s => s.id === 'flight-results');
  assert.ok(flight.runs[2].fidelity.missing.includes('$1,248'));
  console.log(`The missing price failed the gate. Report: ${output}/report.html`);
} finally {
  await Bun.write(path, original);
}

The nested demo command should exit with code 1. The checking script itself should succeed and print “The missing price failed the gate.” Each run uses a new directory under results/, printed by the script, so an old report cannot accidentally satisfy the assertion. Run this experiment on its own so another test is not reading the fixture while it is temporarily changed.

This is a small example of how I want to use an agent: ask it to make an improvement, but give it a way to discover that its improvement has removed something important. The check needs to be able to disagree with the generated output.

The metric can be correct and still answer the wrong question

The fidelity check walks the scenario data, collects string and numeric leaves, normalises the rendered text, and looks for each value. That makes it independent of a generated component’s structure. I do not have to predict which classes or elements a model will choose.

The tradeoff is that it measures presence, not relationships. Here is an intentionally wrong result that still gets full fidelity. Save it as check-fidelity-limit.ts in the same pinned checkout and run it with bun check-fidelity-limit.ts:

import assert from 'node:assert/strict';
import { computeFidelity } from './src/checks.ts';

const data = {
  flights: [
    { airline: 'SWISS', price: '$1,248' },
    { airline: 'United', price: '$986' },
  ],
};
const swappedPrices = 'SWISS $986\nUnited $1,248';
const result = computeFidelity(data, swappedPrices);
assert.equal(result.rate, 1);
assert.deepEqual(result.missing, []);
console.log('Every value is present, but the prices belong to the wrong flights.');

This is a limitation of the implemented check. Verifying the association would require a stronger contract: for example, identifying each flight row and checking its fare within that row. That would buy semantic precision at the cost of needing a reliable way to identify the row across generated layouts.

The aggregate needs care too. uivet averages the fidelity rates across runs. In the missing-price experiment, two runs find all 25 expected values and the third finds 24. The mean is about 98.67%, displayed as 99%. That looks close to perfect, but the requirement was to preserve every value. The default threshold of 1 rejects it. Lowering the threshold would change what the green result promises.

I also keep the individual runs in the report. An average tells me less than seeing which generation lost which value. A model’s occasional failure can matter more to a user than its average score suggests.

Separate observations from release policy

The harness collects more than it enforces. At the pinned revision, its explicit gates cover fidelity, critical accessibility violations, judge score, and judge-score consistency. The latter two are advisory unless enforcement is enabled. Layout and console observations are reported but do not have independent gates in that list.

That distinction is easy to lose in a description such as “checks accessibility, layout, and errors.” Collecting a signal does not mean the exit code rejects it. To use the tool in a release pipeline, I need to read the gate policy as well as the list of checks.

Baseline comparison is a separate decision again. The implementation detects a drop in fidelity and newly appearing accessibility rule IDs relative to a saved baseline. A change can stay above an absolute threshold and still regress from an earlier result. Conversely, refreshing a baseline after a change means accepting that change as the new comparison point; it should be intentional.

This is what interests me about evaluation work. The hard part is deciding what evidence is strong enough for the action that follows. A report can help me investigate a generation without being sufficient to approve a release.

Read what PASS means

The default fidelity threshold requires all expected values to be found. Accessibility is more nuanced: the default gate rejects critical violations, while the report also lists serious ones. My successful offline run still reported serious accessibility issues. A green result therefore does not mean the page is accessible or ready to ship.

The optional model judge is advisory by default. It can draw attention to a design problem, but its opinion does not block the build unless enforcement is enabled. I prefer that separation because a subjective score should not quietly acquire authority over requirements that can be checked directly.

The prototype has other limits. It uses a single 1280 × 800 viewport, so it does not test mobile layouts. Fidelity uses text matching: it can miss a semantic mistake even when a value is present, and formatting can affect a match. The text checks are English-oriented. A fare appearing somewhere on a page does not establish that it is attached to the correct flight.

Those are reasons to read the report and test the interaction too. I want the harness to catch a class of mistakes repeatedly, without pretending it has judged the entire interface for me.