test(perf): no tightening hint for an autoTighten:false baseline

Codex: the per-PR checker still printed "Lower the baseline … with this run
as evidence" for the J3 entry when a run read the low end of its band (731),
the change the flag exists to prevent. Such an entry now passes with the
reason instead, and is not counted as tightenable.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
4grayandClaude Opus 5.5 committed 2026-10-15 08:03:37 +02:00
1 parent 658cab1bf5
commit 443cfa41e4
3 files changed
+45 -1

No files matched your search

+3 -1
View File
@@ -1228,7 +1228,9 @@ that file:
`journeys.<journey>.counters` and a wall-clock entry (one with
`toleranceRatio`) only from `journeys.<journey>.wallClock`, so a value in
the wrong section also counts as missing;
- a measurement below its baseline passes and prints a "tighten" hint;
- a measurement below its baseline passes and prints a "tighten" hint,
except for an entry with `"autoTighten": false`, whose band one low run
does not move;
- a measured counter without a baseline is noted, not failed;
- checking nothing fails: an empty baselines file, or `--only` naming an
entry that does not exist, cannot exit 0;
@@ -12,6 +12,8 @@
* never silently disable the ratchet.
* - A measurement below its baseline prints a "can tighten" hint. Baselines
* are lowered by hand, with the measured output as evidence, never raised.
* An entry with `"autoTighten": false` gets no hint: its slack covers a band
* of normal values, so one low run is no evidence for a lower `value`.
* - Checking nothing is a failure: an empty baselines file, or `--only`
* naming an entry that does not exist, must not exit 0.
* - An optional `note` (a string) says why an entry is enforced, such as a
@@ -208,6 +210,10 @@ export function compareToBaselines({ baselines, summary, only = [] }) {
result.passed.push(
`${label}: ${formatNumber(measured)}${unit} within ${limitText}; uses ${formatNumber(measured - entry.value)} of ${formatNumber(limit - entry.value)}${unit} slack.`
);
} else if (measured < entry.value && entry.autoTighten === false) {
result.passed.push(
`${label}: ${formatNumber(measured)}${unit} within ${limitText}; below the baseline, but autoTighten is false: lower it by hand only when the top of its band fell too (docs/architecture/performance-journeys.md).`
);
} else if (measured < entry.value) {
result.tightenable.push(
`${label}: ${formatNumber(measured)}${unit} is below baseline ${formatNumber(entry.value)} by ${formatNumber(entry.value - measured)}${unit}. Lower the baseline in ${DEFAULT_BASELINES_PATH} with this run as evidence.`
@@ -151,6 +151,42 @@ test('a counter with slack passes up to value + slack and reports the slack used
assert.match(below.tightenable[0], /below baseline 1,000 by 10 bytes/);
});
test('an entry with autoTighten false gets no tightening hint below its value', () => {
const baselines = {
version: 1,
journeys: {
playback: {
'renderer.domMutationsToPlaying': {
value: 733,
unit: 'mutations',
slack: 20,
autoTighten: false,
},
},
},
};
const result = compareToBaselines({
baselines,
summary: {
journeys: {
playback: {
counters: { 'renderer.domMutationsToPlaying': 731 },
},
},
},
});
assert.deepEqual(result.failures, []);
assert.deepEqual(result.tightenable, []);
assert.match(
result.passed[0],
/731 mutations within 753 .*autoTighten is false: lower it by hand/
);
assert.match(
formatResult(result),
/1 baselines checked, 0 can be tightened/
);
});
test('wall-clock entries fail only above value × toleranceRatio', () => {
const within = compareToBaselines({
baselines,