test(performance): add end-to-end Xtream benchmark harness (#1300)

* docs(performance): plan Xtream benchmark * feat(xtream-mock-server): add deterministic 100k fixture * style(xtream-mock-server): apply repository formatting * fix(xtream-mock-server): harden performance fixture data * feat(xtream-mock-server): add performance control plane * docs(performance): correct Xtream capture plan * fix(xtream-mock-server): harden performance controls * fix(xtream-mock-server): harden control lifecycle * feat(performance): add Xtream preload markers * feat(performance): trace Xtream main phases * feat(performance): mark Xtream store publications * feat(performance): trace Xtream database phases * feat(performance): trace Xtream delete cancellation * feat(performance): capture Xtream phase attribution * feat(performance): mark Sources Xtream refresh * test(performance): define Xtream benchmark evidence contracts * test(performance): add Xtream benchmark runner * test(performance): surface failure evidence writes * test(performance): align database read clock * test(performance): preserve capture failure contracts
2026-10-11 19:46:16 -08:00 · 2026-07-28 08:08:07 +02:00
1 parent 5e725a06f3
commit a2fafcfc08
@@ -172,6 +172,54 @@ explicit sum is `ipcStructuredCloneProxyMs`. See
 [M3U Playlist Module Architecture](./m3u-playlist-module.md#initial-url-import-performance-benchmark-electron)
 for the complete cross-process attribution.

+The same request envelope exposes count-only Xtream database phases for these
+operations:
+
+- `DB_GET_CATEGORIES`: `sqlite.categories.read`
+- `DB_SAVE_CATEGORIES`: `normalize.categories`, then
+  `sqlite.categories.write-transactions`
+- `DB_GET_CONTENT`: `sqlite.content.read`
+- `DB_SAVE_CONTENT`: `sqlite.content.category-map-read`, then
+  `normalize.content`, then `sqlite.content.write-transactions`
+- `DB_CLEAR_XTREAM_IMPORT_CACHE`:
+  `sqlite.xtream-cache-clear.write-transactions`
+- `DB_SEARCH_CONTENT`: `sqlite.search.query`, then `normalize.search-rank`
+- `DB_DELETE_XTREAM_CONTENT`:
+  `sqlite.xtream-delete.collect-user-data`, then
+  `sqlite.xtream-delete.write-transactions`
+- `DB_DELETE_PLAYLIST`: `sqlite.playlist-delete.collect-ids`, then
+  `sqlite.playlist-delete.write-transactions`
+
+Read and query phases cover the exact awaited Drizzle query. Normalization
+phases cover the existing synchronous transforms. The content-write phase is
+one aggregate pair around every existing 100-row transaction, cancellation
+checkpoint, and progress callback; it is not an exact measurement of SQLite
+commit time. Cache clear similarly uses one aggregate pair around all existing
+content and category chunk loops and transactions, including their JavaScript
+and autocommit overhead. A zero-category cache clear still emits one pair with
+`itemCount: 0` and performs no extra SQL.
+
+The Xtream-delete collection span includes its existing ordered category,
+favorite, recently-viewed, and content-ID work. Its `itemCount` deliberately
+counts only content and category deletion candidates; favorite,
+recently-viewed, and hidden-category user data is timed but is not added to
+that count. The matching write count uses the same deletion-candidate
+definition. Playlist-delete collection counts every collected favorite,
+recently-viewed, playback-position, download, content, and category ID. Its
+write count adds the final playlist row. Both write spans include every
+existing cooperative checkpoint, 100-row transaction, progress callback, and,
+for playlist deletion, the final playlist-row autocommit; they are not exact
+SQLite commit-time measurements.
+
+Successful end markers carry only row/item counts. Error or cancellation still
+closes the active phase without metadata and preserves the original error.
+Disabled profiling passes no adapter into the operations, so it performs no
+phase-event or metadata-callback allocation. Worker concurrency, SQL, chunk
+sizes, transaction boundaries, progress ordering, and cancellation checkpoints
+are unchanged. `DB_GLOBAL_SEARCH` is deliberately not instrumented: it
+interleaves queries and ranking across sources, so these single-query phases
+would be misleading.
+
 Formal initial-import comparison also requires both request-scoped captures in
 every measured run to have coherent event-loop delay, event-loop utilization,
 and worker-thread CPU values with no unavailable or invalid reason. Summary
@@ -266,6 +314,15 @@ If a worker operation is canceled:
 Cancellation is cooperative and chunk-based. Already committed SQLite batches
 stay committed.

+With the exact `IPTVNATOR_PERF_WORKER_PROFILING=1` opt-in, receipt of a cancel
+for a correlated active request also emits a
+`performance-cancel-received` worker diagnostic containing only safe
+operation/request IDs and its epoch. The functional cancellation flag is set
+first, this receipt remains distinct from the later authoritative
+`cancelled` event, and `DatabaseWorkerClient` ignores it without settling or
+exposing the pending request. Disabled profiling performs no receipt clock or
+transport work, and `DatabaseWorkerClient.cancel()` remains fire-and-return.
+
 ## Renderer Contract

 The preload bridge keeps the existing database methods but adds scoped worker
@@ -6,7 +6,9 @@
 Xtream Codes API protocol. It is used for:

 - **Local development** — run a full portal without a real Xtream subscription
- **E2E testing** — Playwright spins it up alongside the Angular dev server
+- **E2E testing** — Playwright uses it for both web and Electron workflows
+- **Performance investigation** — a locked-down loopback control plane
+  prepares and identifies a fixed synthetic 100k catalog before capture

 ---

@@ -20,6 +22,9 @@ credentialsToSeed(u, p)   ←── deterministic polynomial hash
        │
        ▼
 faker.seed(seed)           ←── all faker calls use same seed per credentials
+        │
+        ├── performance:performance?
+        │          └── index-derived generator (no Faker/runtime clock)
        │
        ▼
  generateCategories()     ←── live / vod / series categories
@@ -43,19 +48,28 @@ Re-requesting with the same credentials returns the exact same data until

 ```
 apps/xtream-mock-server/
-├── project.json                  ← Nx targets: serve (port 3211), serve-with-watch
+├── project.json                  ← Nx serve, watch, lint, and test targets
 ├── public/
 │   └── marketing/                ← committed fictional release artwork PNGs
 ├── tsconfig.json
 └── src/
-    ├── main.ts                   ← Express app bootstrap, all routes wired up
+    ├── main.ts                   ← env validation, HTTP lifecycle, safe logging
    └── app/
+        ├── server.ts             ← side-effect-free Express app factory/routes
+        ├── server-lifecycle.ts   ← bounded, idempotent HTTP shutdown
        ├── scenarios.ts          ← Credential → ScenarioConfig mapping
        ├── data-store.ts         ← Lazy cache, per-credentials generation
+        ├── performance-control.ts ← bounded request lifecycle controller
+        ├── performance-interception-lifecycle.ts ← reset/shutdown generation
+        ├── performance-control-routes.ts ← token-gated control HTTP API
+        ├── performance-control-validation.ts ← strict request validation
+        ├── performance-control.types.ts ← safe manifest/state contracts
+        ├── performance-manifest.ts ← fixed-order catalog hash/counts
        ├── generators/
        │   ├── categories.generator.ts
        │   ├── live.generator.ts   ← Live streams + EPG listings
        │   ├── marketing.generator.ts ← Fictional release screenshot fixture
+        │   ├── performance.generator.ts ← local-only deterministic 100k fixture
        │   ├── vod.generator.ts    ← VOD streams + VodDetails
        │   └── series.generator.ts ← Series items + SeriesInfo
        ├── handlers/
@@ -95,6 +109,134 @@ Response: `{ payload: <data>, action: <action> }`
 This mirrors the backend proxy in `apps/electron-backend` so the same
 Angular service code works in both environments.

+### Performance control endpoint
+
+The control plane is absent by default. It is mounted only when
+`IPTVNATOR_XTREAM_MOCK_CONTROL=1`; startup then requires a non-empty
+`IPTVNATOR_XTREAM_MOCK_CONTROL_TOKEN` and a literal loopback `HOST`
+(`127.0.0.1` or `::1`). Every `/__control/*` request must carry that exact value
+in `x-iptvnator-performance-token`, including `OPTIONS` preflight requests.
+Configuration is validated before the HTTP listener opens. Normal development
+mode preserves the legacy wildcard bind when `HOST` is unset; control mode
+instead defaults to `127.0.0.1` and rejects an explicitly configured
+non-loopback host. The Nx serve targets do not pin `PORT`, so an explicit shell
+value reaches the parser; its no-value default remains `3211`.
+
+Use a dedicated port rather than the normal `3211` E2E server:
+
+```bash
+HOST=127.0.0.1 \
+PORT=3221 \
+IPTVNATOR_XTREAM_MOCK_CONTROL=1 \
+IPTVNATOR_XTREAM_MOCK_CONTROL_TOKEN=local-benchmark-token \
+pnpm nx run xtream-mock-server:serve
+```
+
+The strict control API is:
+
+```text
+POST /__control/prepare
+POST /__control/reset
+POST /__control/barriers
+POST /__control/barriers/:id/release
+POST /__control/delays
+GET  /__control/state
+```
+
+`prepare` accepts exactly `{"scenario":"performance-100k"}` and materializes
+the fixed `performance:performance` fixture before application capture. It
+returns only this manifest:
+
+```json
+{
+    "epoch": 1,
+    "scenario": "performance-100k",
+    "seed": 91001,
+    "counts": {
+        "categories": { "live": 60, "vod": 20, "series": 20, "total": 100 },
+        "items": {
+            "live": 60000,
+            "vod": 20000,
+            "series": 20000,
+            "total": 100000
+        }
+    },
+    "bytes": 123,
+    "catalogSha256": "64-lowercase-hex-characters"
+}
+```
+
+`bytes` is the UTF-8 byte length and `catalogSha256` is the SHA-256 of one JSON
+object whose property order is fixed as:
+
+1. live categories
+2. VOD categories
+3. series categories
+4. live catalog
+5. VOD catalog
+6. series catalog
+
+Credentials, server origins, response envelopes, and EPG/detail caches are not
+part of the hash input.
+
+`reset` accepts exactly `{"mode":"observations"}` or `{"mode":"all"}`.
+Observation reset clears active/held rules, per-identity occurrences, and the
+ledger while retaining the prepared manifest and epoch. All-state reset also
+calls the data-store reset, clears the manifest, and increments the epoch.
+Either mode first advances an internal observation generation and detaches all
+pre-reset response listeners, so a late `finish` or `close` cannot repopulate
+the freshly cleared ledger. Held barrier/delay clients retain the safe `409`
+reset response where possible; unmatched active responses are aborted.
+
+Barrier and delay rules have an ID plus the exact match tuple:
+
+```text
+(epoch, scenario, transport, canonicalAction, categoryId, occurrence)
+```
+
+`transport` is `direct` or `proxy`. Empty actions canonicalize to
+`get_account_info`; the dispatcher's legacy `get_simple_date_table` alias is
+allowlisted. Occurrences are counted per tuple excluding occurrence, so
+parallel requests for different categories cannot consume each other's rule.
+Numeric category IDs must use canonical decimal spelling. Signs, whitespace,
+leading zeroes, non-decimal notation, unsafe integers, and values outside the
+closed scenario ranges are rejected; invalid incoming aliases collapse to the
+single `all` observation identity and cannot expand the bounded state map.
+Rules are one-shot and match before dispatch or JSON serialization. Barrier
+lifecycle is `arrived → blocked → responded|aborted`; delay lifecycle is
+`arrived → delayed → responded|aborted`. Real request abort events release held
+state; a normally consumed request's `close` event is not treated as an abort.
+
+The state response contains only epoch, the safe manifest, active rules, held
+IDs/count, bounded occurrence counts, and bounded lifecycle entries with
+monotonic timestamps. It never stores or returns the token, username, password,
+raw URL/query, request/response payload, titles, or catalog arrays. Unknown
+incoming actions are represented only as `unknown`.
+
+Control JSON is limited to 16 KiB. Bodies reject unknown/missing/wrong-type
+fields. Each observation epoch accepts at most 32 rules, and the serialized
+ledger retains at most 128 entries. Occurrence state has 512 slots: enough for
+the complete closed scenario/action/category/transport identity domain without
+eviction or counter restart; an unexpected overflow fails closed. IDs,
+scenarios, transports, actions, categories, occurrences, and delays
+(`0..5000` ms) use closed validation. Duplicate IDs/matches and already-past
+occurrences fail closed.
+
+The legacy unauthenticated `POST /reset` remains available in normal mode. It
+returns `410` while the control plane is enabled, preventing cache invalidation
+that would leave the controller's prepared manifest stale. Performance runs
+must use the token-authenticated `/__control/reset`.
+
+Barriers and delays exist for deterministic coordination and smoke tests only.
+Formal performance captures must prove that both rule sets are empty and must
+never add an artificial delay to the timed application path.
+
+`SIGINT` and `SIGTERM` use one bounded, idempotent shutdown path. The
+application hook invalidates active observations, settles barriers and delays,
+and clears their timers/listeners before the HTTP listener closes. The shared
+HTTP helper then closes idle and active connections, so an unreleased control
+rule cannot keep the mock process alive.
+
 ### M3U fixture endpoint

 ```
@@ -113,8 +255,12 @@ GET /movie/<username>/<password>/<streamId>.<ext>
 GET /series/<username>/<password>/<streamId>.<ext>
 ```

-All redirect to a publicly available HLS test stream
-(`https://test-streams.mux.dev/x36xhzz/x36xhzz.m3u8`).
+In normal mode all redirect to a publicly available HLS test stream
+(`https://test-streams.mux.dev/x36xhzz/x36xhzz.m3u8`). With the control plane
+enabled, every stream/timeshift route for the performance credentials returns
+`410` without a redirect or outbound request. `/playlist.m3u` is also `410` in
+that mode. This keeps formal performance fixtures local-only; normal
+control-disabled E2E behavior is unchanged.

 ---

@@ -242,19 +388,20 @@ sometimes only respond to that misspelled action.

 ## Scenarios

-| Key (`username:password`) | Seed | Categories               | Items/cat | Account status |
-| ------------------------- | ---- | ------------------------ | --------- | -------------- |
-| `user1:pass1`             | 1001 | 8 each                   | 40        | active         |
-| `large:large`             | 9999 | 20 each                  | 200       | active         |
-| `stress:stress`           | 7777 | 16 each                  | 120       | active         |
-| `series:series`           | 2002 | live:3, vod:4, series:15 | 30        | active         |
-| `minimal:minimal`         | 3003 | 2 each                   | 5         | active         |
-| `epg:epg`                 | 6006 | live:2, vod:1, series:1  | 3         | active         |
-| `emptyvod:emptyvod`       | 7007 | 2 each                   | 5         | active         |
-| `marketing:marketing`     | 8020 | live:4, vod:4, series:4  | curated   | active         |
-| `expired:expired`         | 4004 | 4 each                   | 10        | Expired        |
-| `inactive:inactive`       | 5005 | 4 each                   | 10        | Disabled       |
-| `<any other>`             | hash | 6 each                   | 30        | active         |
+| Key (`username:password`) | Seed  | Categories                 | Items/cat | Account status |
+| ------------------------- | ----- | -------------------------- | --------- | -------------- |
+| `user1:pass1`             | 1001  | 8 each                     | 40        | active         |
+| `large:large`             | 9999  | 20 each                    | 200       | active         |
+| `stress:stress`           | 7777  | 16 each                    | 120       | active         |
+| `performance:performance` | 91001 | live:60, vod:20, series:20 | 1,000     | active         |
+| `series:series`           | 2002  | live:3, vod:4, series:15   | 30        | active         |
+| `minimal:minimal`         | 3003  | 2 each                     | 5         | active         |
+| `epg:epg`                 | 6006  | live:2, vod:1, series:1    | 3         | active         |
+| `emptyvod:emptyvod`       | 7007  | 2 each                     | 5         | active         |
+| `marketing:marketing`     | 8020  | live:4, vod:4, series:4    | curated   | active         |
+| `expired:expired`         | 4004  | 4 each                     | 10        | Expired        |
+| `inactive:inactive`       | 5005  | 4 each                     | 10        | Disabled       |
+| `<any other>`             | hash  | 6 each                     | 30        | active         |

 ### `epg:epg` fixture details

@@ -296,6 +443,91 @@ This scenario is reserved for release screenshots and marketing materials:
 - the SVG renderer in `marketing.generator.ts` remains the fallback for missing
  assets, live logos, season covers, and episode thumbnails

+### `performance:performance` fixture details
+
+This scenario is reserved for local performance captures:
+
+- exactly 60 live categories/60,000 channels, 20 VOD categories/20,000 movies,
+  and 20 series categories/20,000 series
+- exactly 1,000 catalog items in every category
+- fixed IDs, timestamps, ratings, names, and one lazy season/episode generated
+  from indexes rather than Faker, `Date.now()`, or `Math.random()`
+- empty/local-only artwork and direct-source fields
+- VOD and series detail records are materialized only on request
+- preparation and manifest hashing happen before the timed application request
+
+### Electron Xtream performance benchmark
+
+Run a formal benchmark from a clean worktree with the standard CDP endpoint
+available at `127.0.0.1:9222`:
+
+```bash
+perf_output="$PWD/dist/performance/$(date -u +%Y%m%dT%H%M%SZ)-xtream"
+IPTVNATOR_PERF_OUTPUT_DIR="$perf_output" \
+IPTVNATOR_PERF_VARIANT=baseline \
+NX_TASKS_RUNNER_DYNAMIC_OUTPUT=false \
+pnpm nx run electron-backend-e2e:benchmark-xtream --skip-nx-cache
+```
+
+The target starts an isolated control-enabled mock server on
+`127.0.0.1:3221` and uses only the synthetic `performance:performance`
+credentials and fixture. It covers initial import, refresh, deletion,
+cancellation during import, and navigation/search/UI interaction during a
+background operation. Every scenario has one warm-up, five measured
+iterations, and one diagnostic iteration. Only the five measured iterations
+of a clean formal run can have `validForComparison: true`.
+
+Capture setup and operation measurement are separate boundaries. The harness
+first installs and arms the main, renderer, and worker capture infrastructure;
+it then starts the main measurement and renderer operation window immediately
+before triggering the user operation. Profiler attachment, fixture preparation,
+and other setup outside that operation window are not application work. Because
+`database.worker` is persistent but created lazily, the harness awaits the
+read-only `window.electron.dbGetAppPlaylists()` preload call immediately after
+installing main capture. Its result is discarded: this pre-arm only guarantees
+that the exact worker can be profiled and happens before seed or measured
+capture.
+
+Diagnostic profiles are evidence envelopes, not comparison totals. Starting
+Chromium tracing and the CPU profilers can add a short pre-trigger setup segment
+that contains no application workload; the raw boundary timestamps preserve
+that segment explicitly. Formal summaries compare only the five measured runs,
+never warm-up or diagnostic totals.
+
+CPU scopes in the summary are intentionally not interchangeable. Electron main
+CPU uses `process.threadCpuUsage()` and is labeled
+`electron-main-thread`. Database-worker CPU is the sum of thread CPU from valid,
+non-overlapping request work and is labeled
+`sum-valid-request-thread-cpu`; it is neither process-wide CPU nor a whole-worker
+sample.
+
+Scenarios that need an existing portal seed it outside the measured iteration.
+Seed completion is authoritative only after a successful
+`store.xtream-import-terminal` renderer phase marker, the matching terminal DOM
+gate, and two consecutive painted `requestAnimationFrame` frames. The harness
+then verifies main-process settlement before rolling over to the measured
+capture; merely seeing the catalog or loading overlay disappear is not a seed
+terminal.
+
+For an end-to-end wiring check, run smoke mode with one comparison-ineligible
+measured iteration per scenario. An unused loopback CDP port may be selected
+when the formal port is occupied:
+
+```bash
+perf_output="$PWD/dist/performance/$(date -u +%Y%m%dT%H%M%SZ)-xtream-smoke"
+IPTVNATOR_PERF_OUTPUT_DIR="$perf_output" \
+IPTVNATOR_PERF_VARIANT=smoke \
+IPTVNATOR_PERF_SMOKE=1 \
+IPTVNATOR_PERF_CDP_PORT=9322 \
+NX_TASKS_RUNNER_DYNAMIC_OUTPUT=false \
+pnpm nx run electron-backend-e2e:benchmark-xtream --skip-nx-cache
+```
+
+The manifest, summary, per-iteration JSON, traces, CPU profiles, and heap
+snapshots stay under the git-ignored `dist/performance/` tree. Never commit
+those artifacts, and never substitute real credentials, portal URLs, or
+provider data.
+
 ---

 ## Playwright Integration
@@ -337,4 +569,5 @@ await page.route('**/localhost:3000/xtream**', async (route) => {
  `OPENAI_API_KEY=... pnpm release:artwork:generate`, inspect the generated PNGs,
  and finish with `pnpm release:artwork:validate`
 - **Adjust data volume**: Change `itemsPerCategory`, `seasonsPerSeries`, or `episodesPerSeason` per scenario
- **Custom stream URLs**: Edit the HLS stub redirect in `main.ts`
+- **Custom stream URLs**: Edit the normal-mode HLS stub redirect in
+  `app/server.ts`; never enable it for the performance fixture