ci(perf): add the weekly baseline tightening workflow (#1760)

This commit is contained in:
4gray authored and GitHub committed 2026-09-30 18:50:30 +02:00
1 parent 97f56e219c
commit ec8b931dbf
9 files changed
+1142 -58

No files matched your search

+48 -1
View File
@@ -628,7 +628,9 @@ the job for every PR until #1734. Treat the job as blocking before merging;
making it a required check is a maintainer decision.
The runtime counters come from the `Performance journeys` job of the same
workflow, on `ubuntu-latest` only. It runs `pnpm run perf:journeys` under
workflow, on `ubuntu-latest` only. Through the
`.github/actions/performance-journeys` composite action it runs
`pnpm run perf:journeys` under
`xvfb-run` (the Nx target builds `electron-backend:build-performance`, the
Playwright config starts the Xtream mock), writes the measurements to the job
summary and uploads `dist/performance/journeys/` as the `performance-journeys`
@@ -661,6 +663,51 @@ it as `stable: false` because the dashboard flicker it reports is a race
there (see [Settle window](#settle-window)). Add the runner's number once
that flicker is fixed and the counter is deterministic.
### Weekly tightening
`.github/workflows/performance-ratchet.yml` lowers baselines without waiting
for someone to act on a "tighten" hint. Every Monday, and on
`workflow_dispatch`, three `ubuntu-latest` jobs measure the same commit
independently: the production `apps/web` build with
`measure-initial-bytes.mjs --summary`, then the journeys through the
`.github/actions/performance-journeys` composite action, which the
`Performance journeys` job above uses too. A failed journey run does not stop
its job; its entries are then unmeasured in that run. A final job runs
`tools/performance/tighten-baselines.mjs` on the three runs:
- an entry is lowered only when every run measured it and every measurement
is strictly below `value`; the new `value` is the largest of the three (for
a wall-clock entry the largest per-run summary value, which is the largest
P50 for a `.p50` entry);
- a counter marked `counterStability.<name>.stable: false` in any run is
kept, and the report says which run and which iterations disagreed;
- `value` never goes up, `slack` and `toleranceRatio` never change, and no
entry is added or removed: the result must pass
`check-baseline-direction.mjs` without `--allow-increase`, which both the
script and the job check;
- a lowered entry gets `updatedAt`, `measuredWith` and `evidenceRun` (the
workflow run URL); `evidencePr` is set to the tightening PR once it exists.
When the file changed and the run is on `master`, the job pushes
`automation/performance-ratchet` and opens (or updates) a pull request with
the per-run table, the diff and the run URL, labelled `no-release-note`. It
pushes with the existing `PAT` secret, as the Windows MPV pin refresh does,
because a pull request pushed with `GITHUB_TOKEN` starts no CI. Each run
replaces the branch with one fresh commit, except when the open tightening
pull request carries a commit the workflow did not make (a review edit, an
"Update branch" merge): then it leaves the branch alone with a warning, and
the numbers stay in the job summary. When no
baseline was below its value in all three runs, the workflow ends without a
pull request. A dispatch on another branch measures and prints the diff but
never opens one, so `gh workflow run performance-ratchet.yml --ref <branch>`
validates a change to the workflow once the file is on `master`. GitHub only
dispatches workflows that exist on the default branch, so before the first
merge of a new or renamed workflow add a temporary `push` trigger for the
branch and drop it before review, as #1760 did. Review the pull request like a manual
tightening: if `master` moved since the measured commit, the
`Initial bytes ratchet` job on the pull request is what shows that the new
value still holds (the concurrent-merge effect above).
## Charset parse benchmark
V8 stores a string as two-byte UTF-16 once one character falls outside
+6 -1
View File
@@ -260,7 +260,12 @@ bytes on the initial path (the J1 counter `renderer.initialBytes`).
`tools/performance/journey-baselines.json`; baselines only move down. CI runs
the same check in the `Initial bytes ratchet` job of `ci.yml` for PRs that
target `master` and for `master` pushes (dispatch it with
`gh workflow run ci.yml --ref <branch>` for a stacked branch). `perf:journeys` builds the `electron-performance` configuration and runs every
`gh workflow run ci.yml --ref <branch>` for a stacked branch). The weekly
`performance-ratchet.yml` workflow lowers baselines through a bot PR; validate
a change to it with `gh workflow run performance-ratchet.yml --ref <branch>`,
which measures but opens no PR off `master`. Dispatch needs the workflow file
on `master`; before that, see the temporary-trigger note under Weekly
tightening in the performance journeys document. `perf:journeys` builds the `electron-performance` configuration and runs every
journey spec against the Xtream mock: J1 launch, then J2 open-source (a
second set of launches, each followed by the click on the portal card), both
written to the same summary file; its probe specs run with