test(auto-thumbnail): [PKG-AUTO-THUMB-03] isolate native first-frame decoding checkpoint

This commit is contained in:
Harvey Zhao committed 2026-09-14 11:41:33 +08:00
1 parent 772d79a671
commit afb632d16f
9 files changed
+2138 -5

No files matched your search

+2 -1
View File
@@ -132,7 +132,8 @@
"test:scaffold": "node --test test/plugin-scaffold.test.js",
"typecheck:library": "node node_modules/typescript/bin/tsc -p scripts/tsconfig.library.json --noEmit",
"test:library": "node --test refactor/scripts/build.test.mjs test/library-build.test.js test/performance-report.test.js test/plugin-scaffold.test.js test/dev-server.test.js",
"test:dev-server": "node --test test/dev-server.test.js"
"test:dev-server": "node --test test/dev-server.test.js",
"probe:auto-thumbnail-native": "node refactor/scripts/auto-thumbnail-native-probe.mjs"
},
"browserslist": "last 1 Chrome version",
"devDependencies": {
@@ -158,6 +158,26 @@ These variants still fail the unique-first-frame check on the recorded Windows
WebKit host. They are diagnostic page-owned callbacks, not production fixes or
cancellation code. See refactor's first-seek checkpoint before repeating them.
`yarn probe:auto-thumbnail-native` removes ArtPlayer and this plugin entirely.
It serves the frozen timeline with the owned dev server, plus a diagnostic H.264
transcode with B frames disabled, and samples native video/canvas in hidden paused,
visible paused and hidden play-then-immediately-pause modes. It verifies both
files' first purple pixel with FFmpeg before browser capture. FFmpeg/FFprobe must
be installed (optional FFMPEG/FFPROBE executable paths); exact version, arguments,
media hashes, browser versions, events and all31 samples are recorded in an ignored
cache directory. It does not replace the acceptance fixture or add an npm dependency.
The Windows18-case capture still misses the first purple frame in all6 WebKit
cases without any plugin code; Chromium/Firefox controls recover it. The visible
original video can remain at currentTime0 while canvas exposes the later red
frame. This narrows investigation to the native surface on that tested host, but
does not certify all Safari versions or excuse plugin pixel gates. Exploratory
rate0/slow-rate, fastSeek, tiny first-seek offsets and pre-load/metadata layout or
paint waits did not fix it either. The next useful pixel evidence is the same
native/candidate case on another supported backend/device or a decoded-frame
path with explicit timestamps, not another arbitrary wait. See the native-decoder
checkpoint in refactor; AUTO-THUMB-PIXEL-01 and task03 remain open.
The frozen eight-second timeline has one unique purple first frame followed by
red, black, blue and yellow sections. Its command and fingerprint are in
`refactor/baselines/auto-thumbnail-timeline-media.json`. Reproduce into a new
File diff suppressed because it is too large. Load diff
@@ -0,0 +1,72 @@
# PKG-AUTO-THUMB-03: isolate the native decoder boundary
Checkpoint from 772d79a67196ff61dc1e90499c275fee936cb942. Task03 and
AUTO-THUMB-PIXEL-01 remain open; no source fix, completion or publication is claimed.
## New evidence
The earlier first-seek experiments still imported the plugin. This checkpoint
adds `refactor/scripts/auto-thumbnail-native-probe.mjs` and the root command
`yarn probe:auto-thumbnail-native` to remove that dependency. It imports neither
ArtPlayer nor the thumbnail factory: only a native video, canvas, Playwright and
the already-verified local dev HTTP server. The original frozen MP4 remains
byte-identical. A separately generated H.264 variant with B frames disabled probes
whether frame reordering alone explains the observation; it is never substituted
for the original acceptance sample. FFmpeg decoding must show a purple first
pixel in both files before the browser run begins.
The final18 cases cover2 media files,3 browsers and3 native modes: hidden paused,
visible paused and hidden play-then-immediate-pause. Every case records31 draws,
native events, currentTime, readyState, paused/seeking flags and RGBA. Requested
16ms sample timers run slower in Windows WebKit; elapsed times are retained rather
than pretending the nominal half-second is exact. All6 WebKit26.6 cases miss the
purple first frame. Visible original video eventually yields red while time stays0;
other WebKit cases remain transparent. All12 Chromium153.0.8010.12/Firefox155.0
controls obtain purple. This proves the issue can occur without our plugin on
this Windows host; it does not prove the cause in all WebKit/Safari backends.
The initial12-case run omitted visible-paused; its report is retained separately.
The final capture adds visible control and bounds native play Promise waiting.
Results, media metadata, FFmpeg version/arguments, source and raw-report hashes
are in [auto-thumbnail-native-decoder.json](../baselines/auto-thumbnail-native-decoder.json).
Exploratory current-bundle variants also failed: zero/slow playback rate,
fastSeek(0), zero-rate play; first-seek offsets1us/1ms/16ms; layout before src,
two paints before src, layout after metadata and two paints after metadata.
The zero-rate play variant timed out, so it supplies no completed drawing result.
These cache-only experiments are not production cancellation implementations or
accepted fixes. Raw output and script/report hashes are retained for diagnosis.
One exploratory launch failed before browser execution because a data URL could
not resolve the bare Playwright import; executing a real cache .mjs fixed the
runner. No pixel assertion was relaxed to hide these outcomes.
## Source interpretation and next action
Current upstream [Windows Media Foundation implementation](https://github.com/WebKit/WebKit/blob/main/Source/WebCore/platform/graphics/win/MediaPlayerPrivateMediaFoundation.cpp)
starts a native session for seeking and resolves seek completion from the session
started handler, separately from frame painting. That source makes a timing/surface
boundary plausible; it is not the pinned binary's exact source revision and is
not proof of the underlying defect. Runtime captures above are the actual evidence.
Do not add an arbitrary wait, first-time offset or sample replacement to pass this
host's gate. The next useful pixel investigation requires another supported native
backend/device or an explicit decoded-frame path. No change to task03 acceptance,
the first-frame pixel check, public time formula or declared browser support is
authorized by this checkpoint. Other resource/typing/combination work may proceed
without calling this defect fixed.
## Reproduction, ownership and validation
Run under canonical Node24.21.0/Yarn1.22.22 with installed FFmpeg and FFprobe.
Optional FFMPEG/FFPROBE specify executable paths. No new npm dependency, lock or
package artifact change; the probe is explicit, not added to regular CI because
it needs the external encoder binaries. It owns its random-port server, browser
instances/pages and video/canvas cleanup, and retains only its generated diagnostic
directory. Existing8082 remains untouched. It creates no real workspace package.
The command's successful exit means capture completed, not pixel acceptance.
Targeted script lint and plan/risk checks validate only this changed diagnostic and
metadata. Production tests are not rerun because runtime/type/acceptance-test files
are unchanged. The package maintenance guide explains the new diagnostic boundary.
Commit as a task03 checkpoint; revert that commit to remove this diagnostic and
records together. Local only, no push or publication.
+1 -1
View File
@@ -530,7 +530,7 @@
- PKG-AUDIO-05: [记录](changes/2026-09-12-PKG-AUDIO-05-combinations.md) [记录](baselines/audio-combinations-first.json) [记录](baselines/audio-combinations-checkpoint.json) [记录](audio-validation.md)
- PKG-AUTO-THUMB-01: [记录](baselines/auto-thumbnail-release.json) [记录](baselines/auto-thumbnail-contract.md) [记录](changes/2026-09-13-PKG-AUTO-THUMB-01-contract.md) [记录](baselines/auto-thumbnail-contract-validation.json)
- PKG-AUTO-THUMB-02: [记录](baselines/auto-thumbnail-failures.md) [记录](baselines/auto-thumbnail-failures-validation.json) [记录](changes/2026-09-13-PKG-AUTO-THUMB-02-failures.md)
- PKG-AUTO-THUMB-03: [记录](changes/2026-09-13-PKG-AUTO-THUMB-03-lifecycle-checkpoint.md) [记录](baselines/auto-thumbnail-lifecycle-checkpoint.json) [记录](changes/2026-09-13-PKG-AUTO-THUMB-03-hidden-renderer.md) [记录](baselines/auto-thumbnail-hidden-renderer.json) [记录](baselines/auto-thumbnail-timeline-media.json) [记录](changes/2026-09-13-PKG-AUTO-THUMB-03-frame-presentation.md) [记录](baselines/auto-thumbnail-frame-presentation.json) [记录](changes/2026-09-14-PKG-AUTO-THUMB-03-rendering-readiness.md) [记录](baselines/auto-thumbnail-rendering-readiness.json) [记录](changes/2026-09-14-PKG-AUTO-THUMB-03-first-seek.md) [记录](baselines/auto-thumbnail-first-seek.json)
- PKG-AUTO-THUMB-03: [记录](changes/2026-09-13-PKG-AUTO-THUMB-03-lifecycle-checkpoint.md) [记录](baselines/auto-thumbnail-lifecycle-checkpoint.json) [记录](changes/2026-09-13-PKG-AUTO-THUMB-03-hidden-renderer.md) [记录](baselines/auto-thumbnail-hidden-renderer.json) [记录](baselines/auto-thumbnail-timeline-media.json) [记录](changes/2026-09-13-PKG-AUTO-THUMB-03-frame-presentation.md) [记录](baselines/auto-thumbnail-frame-presentation.json) [记录](changes/2026-09-14-PKG-AUTO-THUMB-03-rendering-readiness.md) [记录](baselines/auto-thumbnail-rendering-readiness.json) [记录](changes/2026-09-14-PKG-AUTO-THUMB-03-first-seek.md) [记录](baselines/auto-thumbnail-first-seek.json) [记录](changes/2026-09-14-PKG-AUTO-THUMB-03-native-decoder.md) [记录](baselines/auto-thumbnail-native-decoder.json)
- PKG-VTT-THUMB-01: [记录](baselines/vtt-thumbnail-release.json) [记录](baselines/vtt-thumbnail-contract.md) [记录](changes/2026-09-13-PKG-VTT-THUMB-01-contract.md)
- PKG-VTT-THUMB-02: [记录](baselines/vtt-thumbnail-failures.json) [记录](changes/2026-09-13-PKG-VTT-THUMB-02-failures.md)
- PKG-VTT-THUMB-03: [记录](changes/2026-09-13-PKG-VTT-THUMB-03-resources.md) [记录](baselines/vtt-thumbnail-resources.json) [记录](changes/2026-09-13-PKG-VTT-THUMB-03-parser.md) [记录](baselines/vtt-thumbnail-parser.json)
+14
View File
@@ -1,5 +1,19 @@
# 进度与证据
## Auto Thumbnail 原生首帧边界检查点
新增独立 video/canvas 诊断,完全不加载 ArtPlayer 或插件。冻结原视频不变,另用
FFmpeg生成无B帧对照,确认两文件首帧均为紫色。Windows三引擎/两文件/三模式
共18组:WebKit26.6六组均未得到紫色,Chromium/Firefox十二组得到;可见视频也
会在currentTime0读到后续红帧。此证据将问题缩小到该宿主的原生画面路径,不能
推断所有Safari或标为修复。速率、微小seek偏移和布局/绘制等待的探索同样无效,
没有采用绕过首帧断言的方案。命令、实际FFmpeg/浏览器版本、事件/像素和报告指纹
见[检查点](changes/2026-09-14-PKG-AUTO-THUMB-03-native-decoder.md)和[证据](baselines/auto-thumbnail-native-decoder.json)。
源码/类型/分发与像素验收未改,仅新增诊断脚本和文档;目标lint与计划/风险检查
通过。PKG-AUTO-THUMB-03/AUTO-THUMB-PIXEL-01保持开放,245项/173 done不变。
下一步按原生边界选择其他支持后端/设备取证,同时继续该包的资源与异步边界审查;
不重复追加任意等待。VAST默认行为选择仍待答复。本地检查点提交,无推送/发布。
## MOD-DEV-01 自有开发服务器与关闭流程完成
HTTP、静态资源与 SSE 拆为严格 TS 模块,修复 Servor4.0.2 端口占用却退出0的
+4 -2
View File
@@ -4187,11 +4187,13 @@
"refactor/changes/2026-09-14-PKG-AUTO-THUMB-03-rendering-readiness.md",
"refactor/baselines/auto-thumbnail-rendering-readiness.json",
"refactor/changes/2026-09-14-PKG-AUTO-THUMB-03-first-seek.md",
"refactor/baselines/auto-thumbnail-first-seek.json"
"refactor/baselines/auto-thumbnail-first-seek.json",
"refactor/changes/2026-09-14-PKG-AUTO-THUMB-03-native-decoder.md",
"refactor/baselines/auto-thumbnail-native-decoder.json"
],
"compatibleResolution": "Establish actual decoded-frame readiness for the existing timeline/grid and preserve silent independent extraction; do not waive colored pixel checks as unsupported media. Intrinsic-size hidden/opacity/clip/visible rendering crossed with current/readyState/first-loadeddata-event strategies still misses the unique first frame in all 12 Windows WebKit profiles. Correct Firefox frames retain totalVideoFrames=0, so that counter cannot be a portable readiness gate. Do not reintroduce these unvalidated workarounds. First-seek diagnostics additionally exclude drawing the first loadeddata frame without seek, one/two RAF waits, post-seek RAF waits and verified forward/back seeking on the recorded Windows WebKit host. The warm comparison first had an invalid stale-event completion, then a timeout; final bounded retries verify actual [1,1,0] seeking and still miss purple. Controls retain purple. These are exclusions, not production fixes.",
"closureCriteria": "Candidate source and artifacts generate usable sampled pixels on native browser controls with lifecycle checks; supported Safari and core combinations remain explicit final gates.",
"workspaceState": "The frames module now waits for native presentation and seek completion on capable browsers. Chromium/Firefox source/main/legacy assert all five cells including first purple frame, second red cell, legitimate black and presentation times. Callback-less Windows WebKit first two cells remain diagnostic, later cells asserted; physical Safari/core combinations remain open. 84 native lifecycle/pixel cases passed with explicit held native-frame vs Blob-fallback paths; 19 new Node cases fail 17 ways on prior checkpoint and all 52 current cases pass per candidate. Risk and task03 remain open."
"workspaceState": "The frames module now waits for native presentation and seek completion on capable browsers. Chromium/Firefox source/main/legacy assert all five cells including first purple frame, second red cell, legitimate black and presentation times. Callback-less Windows WebKit first two cells remain diagnostic, later cells asserted; physical Safari/core combinations remain open. 84 native lifecycle/pixel cases passed with explicit held native-frame vs Blob-fallback paths; 19 new Node cases fail 17 ways on prior checkpoint and all 52 current cases pass per candidate. Risk and task03 remain open. Native-only18-case Windows probe also misses the purple first frame in all6 WebKit26.6 cases (hidden/visible paused and immediate play-pause, original and no-B-frame H.264);12 Chromium/Firefox controls recover it. No ArtPlayer/plugin is imported, narrowing but not resolving the native surface boundary. Rate/offset/layout explorations failed and must not be reintroduced as assumed fixes."
},
{
"id": "VTT-THUMB-DIST-01",
@@ -0,0 +1,118 @@
import assert from 'node:assert/strict'
import { execFileSync } from 'node:child_process'
import { createHash } from 'node:crypto'
import fs from 'node:fs'
import path from 'node:path'
import process from 'node:process'
import { chromium, firefox, webkit } from '@playwright/test'
import { startDevServer } from '../../scripts/library/server.ts'
assert.equal(process.version.slice(1), fs.readFileSync('.node-version', 'utf8').trim())
assert.equal(process.env.npm_config_user_agent?.split(' ')[0], 'yarn/1.22.22', 'Use yarn probe:auto-thumbnail-native')
assert.equal(process.argv.length, 2, 'No probe arguments are supported')
const fingerprint = data => createHash('sha256').update(data).digest('hex')
const original = 'test/browser/media/auto-thumbnail-timeline.mp4'
const originalBytes = fs.readFileSync(original)
assert.equal(fingerprint(originalBytes), JSON.parse(fs.readFileSync('refactor/baselines/auto-thumbnail-timeline-media.json')).file.sha256)
const directory = fs.mkdtempSync('refactor/.cache/auto-thumbnail-native-')
fs.writeFileSync(path.join(directory, 'index.html'), '<!doctype html><title>Native decoder diagnostic</title>')
fs.copyFileSync(original, path.join(directory, 'original.mp4'))
const ffmpeg = process.env.FFMPEG || 'ffmpeg'
const ffprobe = process.env.FFPROBE || 'ffprobe'
const args = ['-hide_banner', '-loglevel', 'error', '-n', '-i', original, '-map', '0:v:0', '-an', '-c:v', 'libx264', '-pix_fmt', 'yuv420p', '-bf', '0', '-g', '30', '-crf', '18', '-movflags', '+faststart', path.join(directory, 'no-b-frames.mp4')]
execFileSync(ffmpeg, args, { windowsHide: true })
const ffmpegVersion = execFileSync(ffmpeg, ['-version'], { encoding: 'utf8', windowsHide: true }).split('\n')[0].trim()
const media = ['original.mp4', 'no-b-frames.mp4'].map((name) => {
const file = path.join(directory, name)
const metadata = JSON.parse(execFileSync(ffprobe, ['-v', 'error', '-select_streams', 'v:0', '-show_entries', 'stream=codec_name,profile,time_base,start_time,duration,has_b_frames', '-of', 'json', file], { encoding: 'utf8', windowsHide: true }))
const raw = execFileSync(ffmpeg, ['-v', 'error', '-i', file, '-frames:v', '1', '-vf', 'scale=1:1', '-pix_fmt', 'rgba', '-f', 'rawvideo', 'pipe:1'], { windowsHide: true })
assert.equal(raw.length, 4)
assert(raw[0] > 100 && raw[2] > 100 && raw[1] < 40, 'Both files must actually decode a purple first frame')
return { file, sha256: fingerprint(fs.readFileSync(file)), metadata, ffmpegFirstPixel: [...raw] }
})
const errors = []
const server = await startDevServer({ root: directory, port: 0, onError: error => errors.push(String(error)) })
const results = []
console.log(`Native decoder diagnostics: ${directory}`)
try {
for (const [engineName, engine] of Object.entries({ webkit, chromium, firefox })) {
const browser = await engine.launch()
try {
for (const file of ['original.mp4', 'no-b-frames.mp4']) {
for (const mode of ['paused', 'visible-paused', 'play-pause']) {
const page = await browser.newPage()
try {
await page.goto(server.url)
const result = await page.evaluate(async ({ file, mode }) => {
const video = document.createElement('video')
video.width = 320
video.height = 180
video.muted = true
video.playsInline = true
video.style.cssText = 'position:fixed;visibility:hidden;width:320px;height:180px'
if (mode === 'visible-paused')
video.style.visibility = 'visible'
document.body.append(video)
const canvas = document.createElement('canvas')
canvas.width = 320
canvas.height = 180
const context = canvas.getContext('2d')
const samples = []
const events = []
const started = performance.now()
let timer
const state = () => ({ elapsed: performance.now() - started, time: video.currentTime, readyState: video.readyState, paused: video.paused, seeking: video.seeking })
for (const name of ['loadedmetadata', 'loadeddata', 'canplay', 'play', 'playing', 'pause', 'seeked', 'error'])
video.addEventListener(name, () => events.push({ name, ...state() }))
try {
const ready = new Promise((resolve, reject) => {
video.onloadeddata = resolve
video.onerror = () => reject(new Error(`Media error ${video.error?.code}`))
timer = setTimeout(() => reject(new Error('Native loadeddata timeout')), 8000)
})
video.src = `/${file}`
await ready
clearTimeout(timer)
if (mode === 'play-pause') {
await Promise.race([
video.play(),
new Promise((_, reject) => { timer = setTimeout(() => reject(new Error('Native play timeout')), 8000) }),
])
clearTimeout(timer)
video.pause()
}
for (let index = 0; index < 31; index++) {
context.clearRect(0, 0, 320, 180)
context.drawImage(video, 0, 0)
samples.push({ ...state(), rgba: [...context.getImageData(160, 90, 1, 1).data] })
await new Promise(resolve => setTimeout(resolve, 16))
}
return { samples, events, mediaError: video.error?.code || 0, supportsFrames: typeof video.requestVideoFrameCallback === 'function' }
}
finally {
clearTimeout(timer)
video.pause()
video.removeAttribute('src')
video.load()
video.remove()
canvas.width = canvas.height = 0
}
}, { file, mode })
const purple = result.samples.filter(({ rgba: [r, g, b, a] }) => a === 255 && Math.min(r, b) > 100 && Math.abs(r - b) < 30 && g < 40)
const entry = { engine: engineName, browser: browser.version(), file, mode, purpleSamples: purple.length, ...result }
results.push(entry)
console.log(JSON.stringify({ engine: engineName, file, mode, purpleSamples: purple.length, first: result.samples[0], last: result.samples.at(-1) }))
}
finally { await page.close() }
}
}
}
finally { await browser.close() }
}
}
finally {
await server.close()
assert.deepEqual(errors, [])
assert.equal(fingerprint(fs.readFileSync(original)), fingerprint(originalBytes))
fs.writeFileSync(path.join(directory, 'report.json'), `${JSON.stringify({ task: 'PKG-AUTO-THUMB-03', capturedAt: new Date().toISOString(), node: process.version, platform: process.platform, ffmpegVersion, args, media, results, scope: 'Native video/canvas only; no ArtPlayer/plugin imported. No-seek hidden/visible paused and hidden play-pause controls capture31 draws separated by16ms requested timers; actual elapsed durations are recorded. The no-B-frame transcode is diagnostic only and never replaces the frozen acceptance fixture. Counts report observations, not acceptance or a production fix.' }, null, 2)}\n`)
}
+3 -1
View File
@@ -1801,7 +1801,9 @@
"changes/2026-09-14-PKG-AUTO-THUMB-03-rendering-readiness.md",
"baselines/auto-thumbnail-rendering-readiness.json",
"changes/2026-09-14-PKG-AUTO-THUMB-03-first-seek.md",
"baselines/auto-thumbnail-first-seek.json"
"baselines/auto-thumbnail-first-seek.json",
"changes/2026-09-14-PKG-AUTO-THUMB-03-native-decoder.md",
"baselines/auto-thumbnail-native-decoder.json"
]
},
{