fix(packaging): stop the Linux frame-copy probe timing out on cold sandboxes (#1294)

The packaging verifier bounded `iptvnator_mpv_helper --runtime-probe` with
RUNTIME_PROBE_TIMEOUT_MS (3s) — a constant it shares with the application's own
startup capability gate. Three seconds is a tight budget for a helper that
dlopens libmpv plus EGL/GL/GBM, and the Flatpak profile is closest to that edge
because the helper runs inside the sandbox against its bundled closure: on
#1277 the job failed three consecutive reruns and passed on the fourth with no
code change, while the concurrent master job passed.

Give the verifier its own budget rather than raising the shared one. The app's
probe is a blocking spawnSync on the Electron main process, so a hung helper
must not stall window creation, and a timeout there degrades gracefully to the
native-view fallback. Nothing waits on the packaging probe but the CI job,
which already has its own 120-minute bound, while a premature kill reports a
healthy package as broken.

- PACKAGE_VERIFICATION_PROBE_TIMEOUT_MS (15s) and
  PACKAGE_VERIFICATION_PROBE_MAX_ATTEMPTS (2) join the frozen probe contract;
  RUNTIME_PROBE_TIMEOUT_MS stays at 3s for the application gate.
- runBoundedRuntimeProbe() retries only on ETIMEDOUT, repeating the identical
  bounded launch (same command, args, env, maxBuffer, killSignal) and
  announcing the retry on stderr so a degrading trend stays visible.

Fail-closed behaviour is unchanged. A hard timeout is the one probe outcome
that says nothing about the payload; spawn errors (a missing helper, a wrapper
launched instead of the real ELF), termination by signal, nonzero exits and
malformed or wrong-protocol lines all still fail on the first attempt, and a
helper that keeps hanging still fails once both attempts are spent.

The four new/extended verifier tests cover retry-then-success (asserting the
second launch is identical to the first), exhausted timeouts still rejecting,
four non-timeout verdicts each probing exactly once, and the attempt bound
itself. Setting MAX_ATTEMPTS to 1 fails four of them.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

docs/architecture/embedded-mpv-native.md

@@ -271,6 +271,10 @@ regular development experiment flag) and one process-wide capability decision
 has succeeded. On Linux x64 that decision validates the profile manifest,
 regular-file/access modes, the complete declared bundled closure and hashes,
 then runs `iptvnator_mpv_helper --runtime-probe` with a three-second timeout.
+That short budget belongs to the application gate alone, because it blocks the
+main process and a timeout there degrades to the native-view fallback;
+`tools/packaging/verify-linux-frame-copy-runtime.mjs` runs the same probe under
+a deliberately larger bound described under "Same-Version Desktop Release Gate".
 The probe loads dependencies through the normal ELF loader, initializes an
 idle libmpv client, creates EGL/OpenGL plus mpv render contexts, then
 creates, maps, validates, and destroys a minimal `16x16` shared-memory ring
@@ -951,6 +955,17 @@ packaging passes with `IPTVNATOR_EMBEDDED_MPV_PLATFORM=linux`,
 `IPTVNATOR_REQUIRE_EMBEDDED_MPV=1`, and one exact
 `IPTVNATOR_LINUX_FRAME_COPY_PROFILE`. Each produced artifact is extracted and
 verified, and the x64 helper probe runs in the intended runtime environment.
+That extracted-artifact probe uses its own 15-second bound rather than the
+application gate's three seconds: nothing waits on it but the CI job, which
+already has a job-level timeout, while a premature kill would report a healthy
+package as broken. Cold sandboxes — Flatpak most of all — can spend seconds
+merely loading libmpv plus EGL/GL/GBM. A hard timeout is also the only probe
+outcome that says nothing about the payload, so the verifier repeats it once
+(two attempts in total, announced on stderr) before failing. Every other
+outcome — spawn error, termination by signal, nonzero exit, or a malformed
+protocol line — remains fail-closed on the first attempt, so a wrapper launched
+instead of the real ELF, a missing helper, or a hung helper still fails
+verification.
 The packaged x64 Playwright smoke first runs its fixture-contract target and
 passes Chromium `--ignore-gpu-blocklist` so Mesa llvmpipe can expose WebGL2 in
 CI. That launch-only flag does not bypass any manifest, hash, loader, or helper

tools/embedded-mpv/README.md

@@ -273,6 +273,19 @@ verifies `system`, `portable`, and `flatpak` independently. Every artifact is
 extracted for manifest, mode, package-metadata, ELF-isolation, and helper-probe
 checks. System formats are probed after their declared dependency is installed;
 Snap and Flatpak also require a sandboxed probe where the runner supports it.
+
+`tools/packaging/verify-linux-frame-copy-runtime.mjs` bounds its helper probe at
+`PACKAGE_VERIFICATION_PROBE_TIMEOUT_MS` (15 s), not the application gate's
+`RUNTIME_PROBE_TIMEOUT_MS` (3 s): the short budget exists because the app's
+decision blocks the Electron main process, whereas nothing waits on the
+packaging probe but the CI job's own timeout, and a premature kill would call a
+healthy package broken. A hard timeout is the one outcome that says nothing
+about the payload, so the verifier repeats it up to
+`PACKAGE_VERIFICATION_PROBE_MAX_ATTEMPTS` (2) times with the identical bounded
+launch and announces the retry on stderr. Every other outcome — spawn error,
+signal, nonzero exit, malformed protocol line — stays fail-closed on the first
+attempt, and a helper that keeps hanging still fails once the attempts are
+spent. Both constants live in `runtime-probe-contract.cjs`.
 For a locally installed `--dangerous` Snap, CI explicitly installs and
 connects `mesa-core22` and `gnome-3-28-1804`, verifies both connections, and
 then runs the application-level diagnostic under Xvfb.

tools/embedded-mpv/runtime-probe-contract.cjs

@@ -1,6 +1,29 @@
+// Shared bounds for `iptvnator_mpv_helper --runtime-probe`.
+//
+// The two timeouts differ on purpose, because the two callers pay different
+// costs for waiting:
+//
+// - RUNTIME_PROBE_TIMEOUT_MS bounds the application's own startup capability
+//   decision, a blocking spawnSync on the Electron main process. A hung helper
+//   must not stall window creation, and a timeout there degrades gracefully to
+//   the native-view fallback, so this budget stays short.
+// - PACKAGE_VERIFICATION_PROBE_TIMEOUT_MS bounds the one-shot packaging check
+//   in tools/packaging/verify-linux-frame-copy-runtime.mjs. Nothing waits on it
+//   but CI, which already has its own job-level bound, while a premature
+//   timeout produces a false "broken payload" verdict on a healthy package.
+//   Cold sandboxes — Flatpak most of all — spend seconds just loading libmpv
+//   plus the EGL/GL/GBM stack, so this budget is generous.
+//
+// A hard timeout is also the one probe outcome that says nothing about the
+// payload, so the packaging verifier repeats it up to
+// PACKAGE_VERIFICATION_PROBE_MAX_ATTEMPTS times. Every other outcome — spawn
+// error, signal, nonzero exit, malformed protocol line — remains fail-closed on
+// the first attempt, and a helper that genuinely hangs still fails the check.
 const RUNTIME_PROBE_CONTRACT = Object.freeze({
     RUNTIME_PROBE_MAX_BUFFER_BYTES: 16 * 1024 * 1024,
     RUNTIME_PROBE_TIMEOUT_MS: 3000,
+    PACKAGE_VERIFICATION_PROBE_TIMEOUT_MS: 15000,
+    PACKAGE_VERIFICATION_PROBE_MAX_ATTEMPTS: 2,
 });
 
 module.exports = RUNTIME_PROBE_CONTRACT;

tools/embedded-mpv/runtime-probe-contract.d.cts

@@ -1,6 +1,8 @@
 declare const RUNTIME_PROBE_CONTRACT: Readonly<{
     readonly RUNTIME_PROBE_MAX_BUFFER_BYTES: 16777216;
     readonly RUNTIME_PROBE_TIMEOUT_MS: 3000;
+    readonly PACKAGE_VERIFICATION_PROBE_TIMEOUT_MS: 15000;
+    readonly PACKAGE_VERIFICATION_PROBE_MAX_ATTEMPTS: 2;
 }>;
 
 export = RUNTIME_PROBE_CONTRACT;

tools/packaging/verify-linux-frame-copy-runtime.mjs

@@ -26,8 +26,9 @@ const {
 } = require('./linux-frame-copy-profile.cjs');
 const { resolveLinuxLauncherLayout } = require('./linux-launcher-layout.cjs');
 const {
+    PACKAGE_VERIFICATION_PROBE_MAX_ATTEMPTS,
+    PACKAGE_VERIFICATION_PROBE_TIMEOUT_MS,
     RUNTIME_PROBE_MAX_BUFFER_BYTES,
-    RUNTIME_PROBE_TIMEOUT_MS,
 } = require('../embedded-mpv/runtime-probe-contract.cjs');
 
 const scriptPath = fileURLToPath(import.meta.url);
@@ -1501,6 +1502,56 @@ export function createRuntimeProbeEnvironment({
     return probeEnvironment;
 }
 
+function writeVerifierWarning(message) {
+    process.stderr.write(message);
+}
+
+function isProbeTimeoutResult(result) {
+    const error = result?.error;
+    return Boolean(
+        error && typeof error === 'object' && error.code === 'ETIMEDOUT'
+    );
+}
+
+/**
+ * Executes the bounded packaged helper capability probe.
+ *
+ * A hard timeout is the only retried outcome: it proves nothing about the
+ * payload, and a cold sandbox can spend seconds merely loading libmpv plus the
+ * EGL/GL/GBM stack. Every other outcome — spawn error, signal, nonzero exit,
+ * malformed protocol line — is a real capability verdict and is returned
+ * unchanged after the first attempt. A helper that genuinely hangs still fails
+ * verification once the bounded attempts are exhausted.
+ */
+export function runBoundedRuntimeProbe({
+    probeRunner,
+    helperPath,
+    probeEnvironment,
+    warn = writeVerifierWarning,
+}) {
+    const probeOptions = {
+        encoding: 'utf8',
+        env: probeEnvironment,
+        killSignal: 'SIGKILL',
+        maxBuffer: RUNTIME_PROBE_MAX_BUFFER_BYTES,
+        timeout: PACKAGE_VERIFICATION_PROBE_TIMEOUT_MS,
+        windowsHide: true,
+    };
+    let result = probeRunner(helperPath, ['--runtime-probe'], probeOptions);
+    for (
+        let attempt = 2;
+        attempt <= PACKAGE_VERIFICATION_PROBE_MAX_ATTEMPTS &&
+        isProbeTimeoutResult(result);
+        attempt += 1
+    ) {
+        warn(
+            `Linux frame-copy runtime probe timed out after ${PACKAGE_VERIFICATION_PROBE_TIMEOUT_MS} ms; retrying (attempt ${attempt} of ${PACKAGE_VERIFICATION_PROBE_MAX_ATTEMPTS}).\n`
+        );
+        result = probeRunner(helperPath, ['--runtime-probe'], probeOptions);
+    }
+    return result;
+}
+
 function validateNoEmbeddedMpvNativeArchiveEntries(
     resourceDir,
     asarListPackage
@@ -1542,6 +1593,7 @@ export function verifyExtractedLinuxFrameCopyRuntime({
     elfInspector = defaultElfInspector,
     probeRunner = runVerifierCommand,
     environment = process.env,
+    warn = writeVerifierWarning,
     asarListPackage = listAsarPackageEntries,
 }) {
     const errors = [];
@@ -1625,18 +1677,12 @@ export function verifyExtractedLinuxFrameCopyRuntime({
         nativeDir,
         runtimeMode: profile.runtimeMode,
     });
-    const probeResult = probeRunner(
-        path.join(nativeDir, 'iptvnator_mpv_helper'),
-        ['--runtime-probe'],
-        {
-            encoding: 'utf8',
-            env: probeEnvironment,
-            killSignal: 'SIGKILL',
-            maxBuffer: RUNTIME_PROBE_MAX_BUFFER_BYTES,
-            timeout: RUNTIME_PROBE_TIMEOUT_MS,
-            windowsHide: true,
-        }
-    );
+    const probeResult = runBoundedRuntimeProbe({
+        probeRunner,
+        helperPath: path.join(nativeDir, 'iptvnator_mpv_helper'),
+        probeEnvironment,
+        warn,
+    });
     errors.push(...validateProbeResult(probeResult));
     return errors;
 }

tools/packaging/verify-linux-frame-copy-runtime.test.mjs

@@ -13,6 +13,7 @@ import {
     parseVerifierArguments,
     readLinuxArtifactMetadata,
     readElfArchitecture,
+    runBoundedRuntimeProbe,
     runVerifierCommand,
     validateSystemPackageDependencies,
     validateExtractedSnapMetadata,
@@ -117,14 +118,37 @@ test('uses the shared frozen runtime probe resource contract', async () => {
         runtimeProbeContract.RUNTIME_PROBE_MAX_BUFFER_BYTES,
         16 * 1024 * 1024
     );
+    // The packaging check may wait far longer than the app's blocking startup
+    // decision, and repeats only the outcome that says nothing about the
+    // payload.
+    assert.equal(
+        runtimeProbeContract.PACKAGE_VERIFICATION_PROBE_TIMEOUT_MS,
+        15000
+    );
+    assert.equal(
+        runtimeProbeContract.PACKAGE_VERIFICATION_PROBE_MAX_ATTEMPTS,
+        2
+    );
+    const runtimeProbeContractTypes = fs.readFileSync(
+        runtimeProbeContractTypesUrl,
+        'utf8'
+    );
     assert.match(
-        fs.readFileSync(runtimeProbeContractTypesUrl, 'utf8'),
+        runtimeProbeContractTypes,
         /readonly RUNTIME_PROBE_TIMEOUT_MS: 3000/
     );
     assert.match(
-        fs.readFileSync(runtimeProbeContractTypesUrl, 'utf8'),
+        runtimeProbeContractTypes,
         /readonly RUNTIME_PROBE_MAX_BUFFER_BYTES: 16777216/
     );
+    assert.match(
+        runtimeProbeContractTypes,
+        /readonly PACKAGE_VERIFICATION_PROBE_TIMEOUT_MS: 15000/
+    );
+    assert.match(
+        runtimeProbeContractTypes,
+        /readonly PACKAGE_VERIFICATION_PROBE_MAX_ATTEMPTS: 2/
+    );
 
     const verifierSource = fs.readFileSync(verifierUrl, 'utf8');
     assert.match(
@@ -135,6 +159,14 @@ test('uses the shared frozen runtime probe resource contract', async () => {
         verifierSource,
         /const RUNTIME_PROBE_TIMEOUT_MS\s*=\s*3000/
     );
+    assert.match(
+        verifierSource,
+        /timeout:\s*PACKAGE_VERIFICATION_PROBE_TIMEOUT_MS/
+    );
+    assert.doesNotMatch(
+        verifierSource,
+        /PACKAGE_VERIFICATION_PROBE_TIMEOUT_MS\s*=\s*15000/
+    );
 
     const backendContractsSource = fs.readFileSync(
         backendRuntimeContractsUrl,
@@ -825,7 +857,7 @@ test('validates an x64 system payload and executes one bounded helper probe', ()
             env: { PATH: '/usr/bin' },
             killSignal: 'SIGKILL',
             maxBuffer: 16 * 1024 * 1024,
-            timeout: 3000,
+            timeout: 15000,
             windowsHide: true,
         });
     } finally {
@@ -996,25 +1028,37 @@ test('excludes only Snap template library roots from Electron isolation checks',
     }
 });
 
-test('rejects helper probes terminated by a signal or hard timeout', () => {
-    for (const probeResult of [
+test('rejects helper probes terminated by a signal or exhausted hard timeouts', () => {
+    for (const { probeResult, expectedAttempts } of [
         {
-            status: null,
-            signal: 'SIGKILL',
-            stdout: '',
-            stderr: '',
+            // An external kill is a real capability verdict, not a flake.
+            probeResult: {
+                status: null,
+                signal: 'SIGKILL',
+                stdout: '',
+                stderr: '',
+            },
+            expectedAttempts: 1,
         },
         {
-            error: Object.assign(new Error('spawnSync helper ETIMEDOUT'), {
-                code: 'ETIMEDOUT',
-            }),
-            status: null,
-            signal: 'SIGKILL',
-            stdout: '',
-            stderr: '',
+            // spawnSync reports a hard timeout as ETIMEDOUT alongside the
+            // SIGKILL it sent; the timeout wins, and a helper that keeps
+            // hanging still fails once the bounded attempts are spent.
+            probeResult: {
+                error: Object.assign(new Error('spawnSync helper ETIMEDOUT'), {
+                    code: 'ETIMEDOUT',
+                }),
+                status: null,
+                signal: 'SIGKILL',
+                stdout: '',
+                stderr: '',
+            },
+            expectedAttempts: 2,
         },
     ]) {
         const fixture = createSystemPayload();
+        const warnings = [];
+        let probeCalls = 0;
         try {
             const errors = verifyExtractedLinuxFrameCopyRuntime({
                 resourceDir: fixture.resourceDir,
@@ -1023,19 +1067,171 @@ test('rejects helper probes terminated by a signal or hard timeout', () => {
                 packageDependencies: DEB_SYSTEM_PACKAGE_DEPENDENCIES,
                 elfInspector: validElfInspector,
                 probeRunner() {
+                    probeCalls += 1;
                     return probeResult;
                 },
+                warn(message) {
+                    warnings.push(message);
+                },
             });
             assert.match(
                 errors.join('\n'),
                 /(?:runtime probe terminated by signal SIGKILL|Unable to execute .*ETIMEDOUT)/
             );
+            assert.equal(probeCalls, expectedAttempts);
+            assert.equal(warnings.length, expectedAttempts - 1);
         } finally {
             fs.rmSync(fixture.root, { recursive: true, force: true });
         }
     }
 });
 
+test('retries one timed-out helper probe and accepts a healthy second attempt', () => {
+    const fixture = createSystemPayload();
+    const probeCalls = [];
+    const warnings = [];
+    try {
+        const errors = verifyExtractedLinuxFrameCopyRuntime({
+            resourceDir: fixture.resourceDir,
+            artifactFormat: 'deb',
+            profileName: 'system',
+            packageDependencies: DEB_SYSTEM_PACKAGE_DEPENDENCIES,
+            elfInspector: validElfInspector,
+            probeRunner(command, args, options) {
+                probeCalls.push({ command, args, options });
+                return probeCalls.length === 1
+                    ? {
+                          error: Object.assign(
+                              new Error('spawnSync helper ETIMEDOUT'),
+                              { code: 'ETIMEDOUT' }
+                          ),
+                          status: null,
+                          signal: 'SIGKILL',
+                          stdout: '',
+                          stderr: '',
+                      }
+                    : successfulProbeRunner();
+            },
+            warn(message) {
+                warnings.push(message);
+            },
+        });
+        assert.deepEqual(errors, []);
+        assert.equal(probeCalls.length, 2);
+        // The retry must repeat the identical bounded launch, never a weaker
+        // one.
+        assert.equal(probeCalls[1].command, probeCalls[0].command);
+        assert.deepEqual(probeCalls[1].args, ['--runtime-probe']);
+        assert.deepEqual(probeCalls[1].options, probeCalls[0].options);
+        assert.equal(probeCalls[1].options.timeout, 15000);
+        assert.equal(warnings.length, 1);
+        assert.match(
+            warnings[0],
+            /timed out after 15000 ms; retrying \(attempt 2 of 2\)/
+        );
+    } finally {
+        fs.rmSync(fixture.root, { recursive: true, force: true });
+    }
+});
+
+test('never retries helper probe outcomes that judge the payload', () => {
+    for (const { probeResult, expected } of [
+        {
+            probeResult: {
+                error: Object.assign(new Error('spawnSync helper ENOENT'), {
+                    code: 'ENOENT',
+                }),
+                status: null,
+                signal: null,
+                stdout: '',
+                stderr: '',
+            },
+            expected: /Unable to execute .*ENOENT/,
+        },
+        {
+            probeResult: {
+                status: 127,
+                signal: null,
+                stdout: '',
+                stderr: 'libmpv.so.2: cannot open shared object file',
+            },
+            expected: /runtime probe failed with status 127/,
+        },
+        {
+            probeResult: {
+                status: 0,
+                signal: null,
+                stdout: '{"protocol":1,"usable":true,"libmpv":"0.41.0","renderApi":"egl"}\n{"protocol":1}\n',
+                stderr: '',
+            },
+            expected: /must emit exactly one newline-terminated JSON line/,
+        },
+        {
+            probeResult: {
+                status: 0,
+                signal: null,
+                stdout: '{"protocol":2,"usable":true,"libmpv":"0.41.0","renderApi":"egl"}\n',
+                stderr: '',
+            },
+            expected: /did not return protocol 1 usable EGL capability/,
+        },
+    ]) {
+        const fixture = createSystemPayload();
+        const warnings = [];
+        let probeCalls = 0;
+        try {
+            const errors = verifyExtractedLinuxFrameCopyRuntime({
+                resourceDir: fixture.resourceDir,
+                artifactFormat: 'deb',
+                profileName: 'system',
+                packageDependencies: DEB_SYSTEM_PACKAGE_DEPENDENCIES,
+                elfInspector: validElfInspector,
+                probeRunner() {
+                    probeCalls += 1;
+                    return probeResult;
+                },
+                warn(message) {
+                    warnings.push(message);
+                },
+            });
+            assert.match(errors.join('\n'), expected);
+            assert.equal(probeCalls, 1);
+            assert.deepEqual(warnings, []);
+        } finally {
+            fs.rmSync(fixture.root, { recursive: true, force: true });
+        }
+    }
+});
+
+test('bounds timed-out helper probes to the shared attempt contract', () => {
+    const timeoutResult = {
+        error: Object.assign(new Error('spawnSync helper ETIMEDOUT'), {
+            code: 'ETIMEDOUT',
+        }),
+        status: null,
+        signal: 'SIGKILL',
+        stdout: '',
+        stderr: '',
+    };
+    const warnings = [];
+    let probeCalls = 0;
+    const result = runBoundedRuntimeProbe({
+        probeRunner() {
+            probeCalls += 1;
+            return timeoutResult;
+        },
+        helperPath: '/packaged/native/iptvnator_mpv_helper',
+        probeEnvironment: { PATH: '/usr/bin' },
+        warn(message) {
+            warnings.push(message);
+        },
+    });
+    assert.equal(probeCalls, 2);
+    assert.equal(warnings.length, 1);
+    // The exhausted timeout is reported verbatim so the caller still fails.
+    assert.equal(result, timeoutResult);
+});
+
 test('requires exact Snap graphics layouts and plugs used by the app', () => {
     const root = fs.mkdtempSync(
         path.join(os.tmpdir(), 'iptvnator-verifier-snap-metadata-')