Commit Graph
3 Commits
Author SHA1 Message Date
4gray 59e08fd2d6 feat(embedded-mpv): add Windows frame-copy support (#1175)
Port the embedded mpv frame-copy pipeline to Windows with WGL rendering and named shared memory. Includes packaging validation, platform gates, tests, and architecture documentation.
2026-07-15 21:27:56 +02:00
4grayandClaude Fable 5 7d75d989e8 feat(embedded-mpv): Linux port of the frame-copy rendering engine (headless EGL) (#1171)
* feat(embedded-mpv): Linux frame-copy helper via headless EGL

Port the frame-copy engine's native layer to Linux (PORTING.md items 1-4):

- frame_helper_gl.h: platform GlContext abstraction. macOS keeps the CGL
  path (moved verbatim); Linux acquires an EGL display in order
  surfaceless-Mesa -> default display -> GBM render node, binds a 3.2 core
  desktop-GL context surfaceless (1x1 pbuffer fallback), and hands mpv
  eglGetProcAddress. The helper's own GL calls link against glvnd
  libOpenGL, so no display server is required.
- frame_shm.h: portable frame_shm_now_ns() (CLOCK_MONOTONIC) shared by the
  helper and the reader addon, replacing the macOS-only
  clock_gettime_nsec_np(CLOCK_MONOTONIC_RAW); producer and consumer stay on
  the same clock.
- embedded_mpv_frame_reader.c: real implementation now also on __linux__
  (the code was already POSIX apart from the clock call).
- binding.gyp: OS==linux executable branch for iptvnator_mpv_helper linking
  system libmpv (-lmpv) + EGL/OpenGL/gbm, with rpaths for $ORIGIN/lib and
  the build-time library dir. The in-process addon still does not link
  libmpv - the ban only binds in-process, the helper is out of process.
- build-embedded-mpv.js: system-dev fallback on Linux (LIBMPV_INCLUDE_DIR
  or /usr/include) so a distro libmpv-dev install builds without staging a
  vendored runtime; a pre-set LINUX_NATIVE_LIBRARY_DIR now wins over the
  vendored lib dir.

Verified on Ubuntu 25.04 / i7-1165G7 (Iris Xe): lavfi smoke per PORTING.md
(idle->loading->playing snapshots at 4 Hz, aspect-fit generation bump
g1 1280x720 -> g2 960x720 for a 4:3 source), reader probe 60 fps at
1080p60 with 0 torn reads, clean quit with no leaked processes or shm.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* feat(embedded-mpv): enable the frame-copy engine gates on Linux

Flip the TypeScript side of the Linux port (PORTING.md item 5). A shared
dependency-free predicate, isFrameCopyPlatformSupported() (linux any-arch,
darwin arm64-only), now backs all four gates so they cannot drift:

- main.ts: the persisted Settings toggle promotes to the env flag on Linux
  too (this runs before window creation and controls the sandbox relax).
- EmbeddedMpvNativeService.isFrameCopyEngineActive/isFrameCopyAvailable.
- EmbeddedMpvFrameCopyAdapter.isSupported.

getSupport() ordering: the frame-copy branch moves above the Linux-only
native-engine prerequisites - the X11/Xwayland display-server check and
the system-mpv-on-PATH probe only bind the --wid native engine, while the
frame-copy helper renders offscreen (headless EGL) and links libmpv
itself. createSession() also skips resolving the native window handle for
frame-copy sessions, which the adapter ignores anyway, so native-Wayland
sessions no longer trip the window-handle assertion.

Settings copy: the i18n frame-copy description now says macOS (Apple
Silicon) and Linux in all 18 languages; stale macOS-only doc comments in
the settings/support interfaces updated alongside.

Tests: platform-gate matrix for the adapter (darwin arm64/x64, linux
x64/arm64, win32) and service specs covering Linux activation under
native Wayland, macOS arm64 staying active, macOS x64 staying native, and
the skipped window handle for frame-copy sessions.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* chore(packaging): CI + package guards for the Linux frame-copy helper

- build-and-make.yaml: install libegl-dev/libgl-dev/libgbm-dev on the
  Linux runner (the helper's EGL backend needs them now that the helper
  target builds on Linux), and verify the built helper exists and DOES
  link libmpv - the inverse of the addon's no-libmpv rule, which still
  holds and stays validated.
- electron-after-pack.cjs: strip iptvnator_mpv_helper from packaged Linux
  apps. It links the build host's system libmpv, which end-user systems
  cannot be assumed to have; the support probe treats the missing helper
  as frame-copy-unavailable (dev-build-only engine until the
  bundled-runtime staging milestone).
- frame_helper_gl.h: log the chosen EGL display tier to stderr (the
  adapter mirrors helper stderr), so bring-up problems on exotic setups
  are diagnosable.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* docs(embedded-mpv): document the Linux frame-copy port

- architecture doc: frame-copy section covers Linux (EGL display tiers,
  build deps, package strip), Linux support matrix notes the frame-copy
  exception to the X11 + system-mpv requirements, Linux measured baseline.
- RESULTS.md: Ubuntu 25.04 / i7-1165G7 (Iris Xe) measurement rows via the
  production helper + reader probe; viewport-size claim reproduced.
- PORTING.md: Linux marked done with pointers to what changed; Windows
  remains the open port and its perf gate the open decision.
- CLAUDE.md + tools/embedded-mpv/README.md: platform scope, Linux dev
  build requirements, system-headers fallback, helper strip.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* test(embedded-mpv): commit the Linux frame-copy measurement probe

linux-frame-probe.mjs reproduces the RESULTS.md Linux rows: spawns the
production helper, attaches the frame-reader addon to the announced shm
generation, and reports new-frame fps, copy wall time, produce->copy age,
torn reads and pixel spread. Usage documented in RESULTS.md.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* fix(embedded-mpv): address multi-agent review findings on the Linux port

Confirmed findings (each verified by 3 adversarial reviewers):

- CI would fail to link the helper: -lOpenGL needs the unversioned glvnd
  libOpenGL.so, shipped only by libopengl-dev, which neither the runner
  images nor the previous apt line provide. Added to the workflow and to
  every documented Linux build-dep list.
- The new 'test -x' dist guard could never pass: webpack's dist asset
  copy drops file modes (helper arrives as 0644). The guard is now
  'test -f'; electron-after-pack.cjs restores the execute bit on packaged
  helpers (also fixes packaged-macOS spawns); the support probe now
  requires X_OK, so a mode-stripped helper reads as frame-copy-unavailable
  and falls back to native instead of failing spawn with EACCES.
- The Settings frame-copy toggle was unreachable in exactly the Linux
  states the port targets: the native-Wayland and missing-system-mpv
  unsupported payloads omitted frameCopyAvailable, and toggle visibility
  derives solely from it. Both returns now advertise availability.

Also from review:

- build-embedded-mpv.js keeps the old graceful-skip contract when the new
  system-dev fallback finds libmpv-dev but the GL/EGL/gbm dev stack is
  missing (previously such machines skipped; a hard electron-build
  failure was a regression).
- createSession derives the window-handle skip from the dispatched addon
  instead of re-evaluating the engine gate, so the two cannot disagree.
- The render thread logs the GL renderer string (surfaceless Mesa can
  silently pick llvmpipe on non-Mesa-primary systems; now diagnosable —
  verified 'Mesa Intel Iris Xe' on this machine).
- Specs pin the new semantics: frameCopyAvailable advertised while native
  is unsupported (Wayland / no mpv), frame-copy supported without a
  system mpv, and the handle-skip test disposes its session through the
  owning adapter.
- Docs: PORTING.md file map reflects the frame_helper_gl.h seam for the
  Windows porter; helper-strip removal correctly gated on milestone 4
  (bundled libmpv), not milestone 3; RESULTS.md preamble notes the
  RAW->MONOTONIC clock change; stale '(macOS)' scope comments updated.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* fix(embedded-mpv): address Greptile/Codex review comments

- Sandbox gate requires a usable helper (Greptile P1, security): the
  main.ts env promotion now also probes for an executable
  iptvnator_mpv_helper before relaxing the window sandbox — a stale
  opt-in on packaged Linux (helper deliberately stripped) or after a
  cleaned native build no longer costs a sandboxless launch for an
  engine that cannot activate. Helper discovery (addon candidate paths +
  X_OK probe) moved into embedded-mpv-frame-copy-platform.util.ts,
  shared by main.ts and the service; the service keeps thin instance
  wrappers so tests can stub per scenario. New util spec pins the
  platform matrix, candidate resolution, and the execute-bit semantics.
- Stale frame-copy artifacts on skipped builds (Codex P2): cleanOutput()
  now also removes iptvnator_mpv_helper and
  embedded_mpv_frame_reader.node, so a failed/skipped rebuild cannot
  leave a previous helper advertising frame-copy support against a
  runtime the build just declared unavailable.
- Multiarch default lib dir (Greptile P1, partially refuted): -l
  resolution never depended on our -L (the compiler's built-in search
  paths include the Debian/Ubuntu multiarch dir — proven by the green CI
  run linking with a nonexistent -L dir), but the system-dev fallback
  now defaults to /usr/lib/<multiarch-triple> when present so the -L
  flag and the helper's baked rpath point somewhere real.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* fix(embedded-mpv): harden Linux frame-copy port

---------

Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
2026-07-15 20:42:50 +02:00
4grayandClaude Fable 5 26271fc076 feat(embedded-mpv): frame-copy rendering engine (experimental, macOS Apple Silicon) (#1169)
* spike(embedded-mpv): frame-copy pipeline prototype (helper + shm ring + Electron viewer)

Standalone macOS spike for the frame-copy unification direction from the
2026-07-10 analysis: a helper process renders mpv offscreen into a GL FBO,
reads frames back through an async PBO ring, and publishes BGRA frames into
a 3-slot POSIX shm seqlock ring; a minimal Electron viewer copies the newest
frame via a plain-C N-API addon and uploads it to a WebGL canvas per rAF.

First numbers on M1 Pro (see spike README): 4K60 HEVC hwdec sustained at
60 fps end to end, ~1.2 ms shm copy + ~3.5 ms texture upload, ~10 ms
produce-to-upload age, zero torn frames. Remaining gates: weak hardware,
long-run pacing, HDR, latency flash test.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* spike(embedded-mpv): add RESULTS.md measurement log with M1 Pro baseline

Structured per-machine table with repro commands so the pending Intel Mac
and Windows iGPU runs can be appended and compared one-to-one.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* spike(embedded-mpv): pacing/judder instrumentation + 50/25 fps and HDR gate results

Viewer now measures inter-frame intervals on both clocks (present side and
producer side): stddev/p99/max, late-frame counters vs the producer's median
interval, and a cumulative LONGRUN summary every 30 s. The addon exposes the
producer timestamp (produceMs) for this.

Measured on M1 Pro: 50 fps and 25 fps cadences are clean (late frames only
at startup; residual jitter is 120 Hz rAF grid quantization, bounded by one
display tick), and 4K25 HDR10 PQ/BT.2020 is tonemapped to SDR by mpv before
readback at full rate with unchanged copy costs. RESULTS.md carries the
tables and HDR-clip repro commands.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* spike(embedded-mpv): record 10-minute 4K60 HEVC long-run results

Zero dropped frames and zero torn reads after the first-minute warmup over
~8.5 minutes; steady-state late frames (~0.4%) track the 12 s test clip's
--loop restarts, not the copy pipeline.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* spike(embedded-mpv): viewport-scaling measurement + integration design draft

Confirms the render-at-viewport-size claim (4K source in a 720p FBO costs
720p: 0.17 ms readback / 0.16 ms copy / 0.17 ms upload at 60 fps) and adds
DESIGN.md — the draft integration architecture: per-session helper process
linking bundled libmpv on all platforms (finally full-featured + Wayland-
agnostic Linux), JSON-over-stdio control evolving the Linux wid protocol,
unchanged EmbeddedMpvSession renderer contract, shm generations for resize,
packaging via the existing vendored-runtime tooling, rollout behind its own
flag with the docked path as default.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* spike(embedded-mpv): auto-detect Homebrew prefix and Node headers for Intel Macs

BREW_PREFIX was hardcoded to /opt/homebrew (Apple Silicon) and NODE_INC to
one nvm version; both now resolve via brew --prefix and the PATH node's
execPath, so the pending Intel Mac run needs no Makefile edits. README gets
a fresh-machine checklist.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* spike(embedded-mpv): self-contained measurement bundle for machines without Node/pnpm

make-bundle.sh assembles a tarball with the spike sources, vendored N-API
headers (Makefile prefers them when present, so no Node install is needed),
pre-generated 4K HEVC/HDR10 test clips, and an official Electron dist
download for the target arch. collect-results.sh builds and runs the full
RESULTS.md scenario suite automatically (plus an optional --long 10-minute
run) and writes one results-<host>-<date>.txt to send back. Target-machine
prerequisites shrink to Xcode CLT + brew mpv — built for the pending Intel
Mac baseline run.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* spike(embedded-mpv): support MacPorts libmpv and legacy-macOS bundles

Makefile and collect-results.sh now detect libmpv in the Homebrew prefix or
MacPorts /opt/local (Homebrew is unsupported on legacy macOS like High
Sierra; 'sudo port install mpv +libmpv' provides libmpv there). make-bundle
takes ELECTRON_VERSION/BUNDLE_SUFFIX overrides — Electron 27+ needs macOS
10.15, so High Sierra bundles ship Electron 26.6.10 (LSMinimumSystemVersion
10.13).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* spike(embedded-mpv): scope macOS frame-copy engine to Apple Silicon only

Owner decision 2026-07-10: skip Intel Mac measurements and gate the future
frame-copy engine on arm64. Intel Macs able to run the app at all are a
shrinking 2015-2020 cohort and keep the docked/external/web player paths;
the macOS hardware gate closes with the M1 Pro numbers, and remaining
hardware risk moves to the Windows/Linux ports.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* feat(embedded-mpv): frame-copy helper process and shm frame reader (native layer)

iptvnator_mpv_helper: one-process-per-session libmpv host that renders
offscreen at viewport size (headless CGL + async PBO ring, validated in
spikes/mpv-frame-copy), publishes BGRA frames into a seqlock shm ring with
resize generations, plays audio directly, and speaks a stdio protocol —
tab-separated commands in, JSON events out. The snapshot event mirrors
NativeEmbeddedMpvSessionSnapshot; status semantics (END_FILE reasons,
eof-reached with keep-open, pause gated on loaded path, fatal-only status
flips) are ported from embedded_mpv.mm.

embedded_mpv_frame_reader.node: plain-C N-API reader the preload script
uses to memcpy the newest complete frame into a V8 ArrayBuffer (Electron's
memory cage forbids zero-copy). Stub exports off macOS.

Both build as extra binding.gyp targets through build-embedded-mpv.js; the
helper gets the same libmpv dependency-path rewrite + ad-hoc re-sign as the
addon and is validated by the forbidden-link check.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* feat(embedded-mpv): frame-copy engine wiring in main process and preload

EmbeddedMpvFrameCopyAdapter implements the NativeEmbeddedMpvAddon surface
over a per-session helper process (spawn, stdio protocol, snapshot cache,
graceful quit->SIGTERM->SIGKILL teardown), so EmbeddedMpvNativeService
reuses its polling/diff/power-blocker/recording logic unchanged. The
IPTVNATOR_ENABLE_EMBEDDED_MPV_FRAME_COPY flag (darwin/arm64 only) routes
getAddon() to the adapter and reports engine: 'frame-copy' in support.

The preload frame pump loads the shm reader addon, copies the newest frame
once per rAF into a reused buffer, and uploads it to WebGL2 on the
renderer's canvas — no frame data crosses the contextBridge; the bridge
only gains attachEmbeddedMpvFrameView/detachEmbeddedMpvFrameView. The
experiment flag relaxes the window sandbox for that native require;
contextIsolation and nodeIntegration:false stay on.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* feat(embedded-mpv): frame-copy canvas mode in the player component + docs

EmbeddedMpvPlayerComponent renders <canvas data-embedded-mpv-frame> when
support reports engine 'frame-copy' and the session controller starts/stops
the preload frame pump around the session lifecycle. The bounds provider
skips HIDDEN_BOUNDS and the popover cutout for this engine — the canvas is
ordinary DOM, dialogs and popovers stack above it natively; bounds sync
still drives the helper's render size. Adapter unit tests cover spawn args,
snapshot caching, shm generations, protocol encoding, unexpected-exit
mapping, and dispose escalation. Architecture doc and CLAUDE.md describe
the engine, its flag, and the sandbox trade-off.

Verified end to end in the built app (M1 Pro): engine detection, helper
spawn, lavfi playback onto the canvas via CDP-injected smoke — including an
orientation fix (helper FLIP_Y already yields texture-order rows; the pump
shader must not flip uv again).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* fix(embedded-mpv): close helper stdin on dispose + lifecycle logging

Live testing surfaced a stray idle helper that survived a session switch;
until the root cause is pinned down, dispose now also closes the child's
stdin (the helper exits on EOF) as a second kill path besides quit ->
SIGTERM -> SIGKILL, and spawn/dispose/exit are logged with the session id
so leaks are attributable.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* fix(embedded-mpv): reap sessions when the renderer reloads or crashes

Root cause of the stray idle helper found during live testing: session
teardown lives in the renderer's Angular lifecycle, which never runs on a
renderer crash or hard reload — the main process kept the session (and its
frame-copy helper process / native mpv handle) alive until app shutdown.
EmbeddedMpvNativeService now watches the main window's webContents for
render-process-gone and did-navigate (full reloads only; in-app Angular
routing emits did-navigate-in-page) and disposes every session. Applies to
both engines. Verified live: location.reload() during frame-copy playback
logs 'Disposing 1 session(s): renderer reloaded' and the helper exits
cleanly. Regression test drives both events against the service.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* feat(embedded-mpv): Settings toggle for the frame-copy engine

New 'Embedded MPV: frame-copy engine' checkbox in Settings > Playback,
shown only when the machine can run it (macOS arm64 with the helper binary
present — support now reports frameCopyAvailable). The choice persists to
the main-process config store because the engine relaxes the window sandbox
for the preload frame pump, which is fixed at window creation: main.ts
reads the store before creating the window and sets the engine env var; an
explicitly set env var (including '0') always wins, and the UI shows a
restart hint while the saved choice differs from the active engine.
Localized in all 18 locales.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* feat(embedded-mpv): aspect-fit rendering in the frame-copy helper

The helper now observes dwidth/dheight and renders its FBO at the
aspect-fit size of the video inside the requested viewport, bumping a shm
generation on change — letterbox bars are never baked into frames (the VOD
watch shell's ~2:1 box no longer shows black side bars; the canvas
background is transparent so the sides show the app surface, while
fullscreen keeps its black backdrop). Frames also get smaller than the
viewport when aspects differ, trimming copy cost. Aspect override changes
refit automatically. Snapshots now carry videoWidth/videoHeight, and the
adapter forwards IPTVNATOR_EMBEDDED_MPV_AUDIO_DELAY to mpv's audio-delay
for lip-sync tuning until proper calibration lands.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* feat(embedded-mpv): require frame-copy artifacts in macOS package validation + docs

macOS packages that ship embedded_mpv.node must also ship the
iptvnator_mpv_helper binary and the embedded_mpv_frame_reader.node addon —
they come out of the same binding.gyp run, and a package missing them would
silently lose the frame-copy engine. Covered in the package-identity test.
Architecture doc and CLAUDE.md document the Settings toggle, aspect-fit
rendering, audio-delay passthrough, and the renderer-reload session reaping.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* fix(electron): inline TS helpers so the sandboxed preload keeps working

The frame pump's async/await (target es2015 + importHelpers) made webpack
externalize tslib in main.preload.js. Sandboxed preloads can only require
Electron's built-in module whitelist, so the entire preload script failed
to load and window.electron disappeared for every run without the
frame-copy flag. importHelpers:false for electron-backend keeps the preload
bundle self-contained — and future async code in preload can no longer
silently reintroduce the breakage. Verified live: sandboxed run now has the
bridge, reports engine 'native' and frameCopyAvailable true.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* docs(embedded-mpv): commit the frame-copy analysis handoff + source inventory

The 2026-07-10 analysis that led to this branch now lives next to the spike
(spikes/mpv-frame-copy/ANALYSIS.md), and the architecture doc's What To
Commit section lists the frame-copy engine sources.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* fix(embedded-mpv): address review findings on the frame-copy engine

- Stale pump attach can no longer win over a newer session: attach/detach
  bump a shared epoch and async attach waits re-check it after every await,
  so an attach for a replaced session aborts instead of installing itself
  (greptile P1).
- A failed frame-view attach (no canvas, no WebGL2, reader missing) now
  disposes the session and surfaces the error UI instead of leaving audio
  playing behind a black canvas (codex P2).
- A stale frame-copy opt-in without the helper binary falls back to the
  native engine instead of reporting embedded MPV unsupported, and the
  Settings checkbox stays visible while a saved opt-in exists so it can
  always be cleared (codex P2). Regression test covers the fallback.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* fix(packaging): make the darwin frame-copy packaging test host-agnostic

On non-macOS CI hosts validatePackagedEmbeddedMpv also reports that macOS
link validation needs a macOS host, so the success-path assertion now
checks only the frame-copy artifact requirement instead of expecting an
empty error list.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* docs(embedded-mpv): Windows/Linux porting handoff for the frame-copy engine

Self-contained entry point for porting sessions on other machines: current
state and coordination constraints, per-OS task lists (Linux EGL first,
then Windows WGL + named shm — the decisive iGPU perf gate), the
hard-won gotchas from the macOS integration (preload/tslib sandbox
breakage, V8 memory cage, frame orientation, stale-attach epoch, dispose
escalation, node-gyp naming, snapshot protocol semantics), testing
recipes, and the suggested milestone order.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* docs(embedded-mpv): branching and merge strategy in the porting handoff

Port work goes to stacked branches off the frame-copy branch (PR base =
frame-copy branch, sequential merges, stack depth one), never into the
frozen PR #1169 branch itself.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* docs(embedded-mpv): drop stale uncommitted note from porting handoff

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* fix(embedded-mpv): harden frame-copy helper startup

---------

Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
2026-07-15 19:48:07 +02:00