test(asr): [PKG-ASR-05] verify video-only input and installed capture recovery

This commit is contained in:
Harvey Zhao committed 2026-09-15 14:07:40 +08:00
1 parent 65368809ee
commit 3c89b48514
9 files changed
+12560 -4

No files matched your search

+19 -2
View File
@@ -154,7 +154,7 @@ From the repository root:
Windows Playwright WebKit 26.6 lacks AudioContext and AudioWorkletNode. Its native
ASR cases remain explicit skips and do not establish Safari/device compatibility.
Native direct/fallback routing, volume/mute, CORS and audio-track combinations
have scoped browser evidence. Unsupported media, external-owner integration,
have scoped browser evidence. Other unsupported media,
Safari and physical devices remain PKG-ASR-05 acceptance work. CORS restrictions
and main-video-only capture are explicit integration boundaries, not a claim
that every media source can be recognized. Do not use the demo's external
@@ -162,10 +162,27 @@ recognizer service as a prerequisite for local audio verification.
## Shared installed browser validation
Six browser files select verified installed ASR bytes through browser-candidate.js. The Audio Track combination independently verifies both installed plugins. Published ASR controls retain their frozen archives; attachments distinguish selected published, installed and source inputs. Tests cover native local PCM, direct/capture routing, CORS, volume/mute and ownership. Deliberately forced Firefox binding rejection remains labeled; no recognition service or physical speaker output is claimed.
Seven browser files select verified installed ASR bytes through browser-candidate.js. The Audio Track combination independently verifies both installed plugins. Published ASR controls retain their frozen archives; attachments distinguish selected published, installed and source inputs. Tests cover native local PCM, direct/capture routing, CORS, video-only input recovery, volume/mute and ownership. Deliberately forced Firefox binding rejection remains labeled; no recognition service or physical speaker output is claimed.
Use `yarn test:package --browser`, then `yarn test:browser:installed`. The common
roster in scripts/browser-validation/scope.ts drives both preparation and required
inputs. Explicit artifact overrides are rejected when an installed map is present;
missing or stale inputs fail without source fallback. Source mode retains deliberate
artifact selection. See ../../refactor/changes/2026-09-15-CI-01-asr-cast-installed.md.
## Video-only input
`test/browser/asr-no-audio.spec.js` uses the generated H.264 video with no audio
track, verifies changing decoded pixels and the media clock, then changes the same
player to the local AAC tone. The default direct graph can yield silence or no
chunks for the video-only source and remains available for the later audio source.
Capture-only mode naturally fails `createMediaStreamSource` with no audio track;
the failed graph closes its context and stops its captured video track without
taking over or pausing native playback. A later playable source reacquires capture.
Both paths verify nonzero recovered PCM and final context/track cleanup.
The test observes real browser nodes/streams and does not inject a binding failure.
It does not prove speaker output, malformed-media recovery, Safari or device
support. Run `yarn test:browser asr-no-audio.spec.js --workers=1` for source mode;
the same file is included in the installed CI roster. WebAudio capability skips
remain visible and cannot be counted as audio processing passes.
File diff suppressed because it is too large. Load diff
@@ -0,0 +1,64 @@
# PKG-ASR-05 无音轨视频与恢复检查点
来源 HEAD:`65368809ee70803e4dae7cb145a17310a386b4e4`。任务继续 doing,
本次补齐明确列出的无音轨场景,不关闭 ASR 的设备、其他不支持媒体或发布要求。
生产源码、类型、依赖、根锁和工作区分发字节均未修改。
## 验证边界
新增 `test/browser/asr-no-audio.spec.js`,使用同一旧核心/候选核心宿主,分别调用
默认 direct 和显式 `audioInput: { type: 'capture' }`。原生 AudioContext 方法仅
记录输入及返回资源,仍调用浏览器实现;不注入异常或伪造 stream/PCM。
`ffprobe` 确认既有 pattern.mp4 只有一条 H.264 视频流,无音轨;后续 audio-tone.m4a
只有一条 48 kHz 单声道 AAC。两份文件未重新生成,完整字节摘要及 probe 输出
保存在[机器证据](../baselines/asr-no-audio-validation.json)。输入变化需要重新验收。
每个可处理音频的场景均验证:
- 无音轨视频的媒体时钟和 Canvas 读取的解码画面持续变化,视频未被暂停或报错,
音量/静音不变,ASR 不产生非零 PCM。
- 默认路径保留原生 element source 和 Context。Chromium 输出全零块;Firefox
此阶段没有回调。二者都是记录到的实际行为,没有强制统一为某种 callback 数量。
- 显式捕获得到仅含一条视频轨的原生 stream,原生 stream source 初始化失败;
Context 关闭、捕获轨停止,零直接绑定,原视频继续播放。初始化错误保留在证据中。
- 同一播放器切换到 AAC 后出现至少三块非零 PCM。默认路径复用一个 Context;
捕获模式新建 Context,旧 Context 已关闭。销毁后全部 Context/捕获轨关闭,DOM 清理。
这不是物理扬声器输出、跨域授权绕过或完整设备能力证明。此前显式跨域失败与同源
恢复已有 ASR-09 证据;本次没有重跑它来制造新进度,ASR-CORS-01 的默认限制仍保留。
## 产物与 CI
`yarn test:package --include=artplayer-plugin-asr` 在 `run-NH0n0v` 快照重建、打包、
仓库外安装并冻结复装 core/chapter/ASR,检查文件摘要。通用 36 项 runtime、
5 组旧类型和 8 组精确类型仍只覆盖 core/chapter,不冒充 ASR 全包消费者验收。
新增测试通过 `browserCandidate` 核验安装文件与源码/构建输入。main 和 legacy
各用同一安装目录内对应真实文件,legacy 映射也逐项符合 tarball 摘要。没有用
源码回退代替安装产物。公开版本尚未升级为下一 major;这只是内部验收候选。
`scripts/browser-validation/scope.ts` 纳入该文件,ASR 安装清单从六个变为七个。
已有清单测试保护该用例不能遗漏;没有修改 CI 并发、超时、重试或其他必需命令。
远端 CI 尚未在本轮执行,局部通过不代表完整二十包矩阵通过。
## 结果与维护
Node 24.21.0、Yarn 1.22.22。源码、installed main、installed legacy 各 8 通过、
4 WebAudio 能力跳过,合计 24 通过、12 跳过、0 失败、0 重试。实际处理来自
Chromium 153.0.8010.12 和 Firefox 155.0;Windows WebKit 26.6 缺 AudioContext/
AudioWorklet,跳过在报告中明确保留,不视为 Safari 验收。
每轮另记录八条 Chromium 媒体 `ERR_ABORTED`,原始诊断保留;用例通过不表示
全部网络请求都正常结束。
ASR 229 项专项及浏览器清单 6 项通过,docs-tools 类型、定向只读 lint 和严格
工具链检查通过。初始 lint 的多变量同一行已修正;运行行为和断言未改变。
复跑:`yarn test:browser asr-no-audio.spec.js --workers=1`。使用安装内容时先运行
上述 package 命令,再把 `ARTPLAYER_BROWSER_ARTIFACTS` 指向其 `browser-artifacts.json`。
全量安装 CI 仍按 `yarn test:package --browser` 与 `yarn test:browser:installed` 执行。
包内 [ARCHITECTURE](../../packages/artplayer-plugin-asr/ARCHITECTURE.md) 同批更新。
本批没有发现新的生产缺陷。回退只撤销新增测试、CI 清单项与文档/证据,不恢复任何
旧运行时错误。后续继续其他不支持媒体、实际设备和完整分发验收;202/266 完成数
不变。没有推送、部署或发布。
+1 -1
View File
@@ -597,7 +597,7 @@
- PKG-ASR-02: [记录](changes/2026-09-13-PKG-ASR-02-audio-baseline.md) [记录](baselines/asr-audio-validation.json)
- PKG-ASR-03: [记录](changes/2026-09-13-PKG-ASR-03-audio-ownership.md) [记录](baselines/asr-ownership-validation.json)
- PKG-ASR-04: [记录](changes/2026-09-13-PKG-ASR-04-public-types.md) [记录](baselines/asr-types-validation.json) [记录](baselines/asr-type-diagnostics.json)
- PKG-ASR-05: [记录](baselines/asr-volume-validation.json) [记录](baselines/asr-fallback-validation.json) [记录](baselines/asr-core.json) [记录](baselines/asr-local-core-validation.json) [记录](changes/2026-09-13-PKG-ASR-05-local-core-checkpoint.md) [记录](baselines/asr-combinations-validation.json) [记录](changes/2026-09-13-PKG-ASR-05-cors-combinations.md) [记录](baselines/asr-explicit-capture-validation.json) [记录](changes/2026-09-13-PKG-ASR-05-editor-generation.md)
- PKG-ASR-05: [记录](baselines/asr-volume-validation.json) [记录](baselines/asr-fallback-validation.json) [记录](baselines/asr-core.json) [记录](baselines/asr-local-core-validation.json) [记录](changes/2026-09-13-PKG-ASR-05-local-core-checkpoint.md) [记录](baselines/asr-combinations-validation.json) [记录](changes/2026-09-13-PKG-ASR-05-cors-combinations.md) [记录](baselines/asr-explicit-capture-validation.json) [记录](changes/2026-09-13-PKG-ASR-05-editor-generation.md) [记录](changes/2026-09-15-PKG-ASR-05-no-audio.md) [记录](baselines/asr-no-audio-validation.json)
- PKG-ADS-01: [记录](baselines/ads-release.json) [记录](baselines/ads-contract.md) [记录](baselines/ads-contract-validation.json) [记录](changes/2026-09-12-PKG-ADS-01-contract.md) [记录](scripts/ads-contract.test.mjs)
- PKG-ADS-02: [记录](changes/2026-09-12-PKG-ADS-02-tests.md) [记录](ads-validation.md) [记录](baselines/ads-validation.json)
- PKG-ADS-03: [记录](changes/2026-09-12-PKG-ADS-03-lifecycle.md) [记录](ads-validation.md) [记录](baselines/ads-lifecycle-validation.json)
+13
View File
@@ -1,5 +1,18 @@
# 进度与证据
## PKG-ASR-05 无音轨视频与恢复
新增真实无音轨 H.264 → AAC 的默认/显式捕获测试:播放时钟与解码画面继续,
无非零 PCM;显式捕获失败后关闭 Context/捕获轨且不直接绑定媒体;切源后恢复
非零采集,销毁清理通过。源码、隔离安装 main/legacy 合计 24 通过、12 Windows
WebKit WebAudio 能力跳过,零失败/重试。229 ASR 专项、6 清单测试、类型/lint/
严格工具链通过。新增文件已纳入 installed CI 清单,远端未执行。
见[检查点](changes/2026-09-15-PKG-ASR-05-no-audio.md)和
[证据](baselines/asr-no-audio-validation.json)。生产运行字节/类型/依赖/锁未改;
任务05仍 doing,不关闭其他不支持媒体、真实设备和发布门槛。202 done、21 doing、
43 todo;继续剩余实际验收,不重复已有跨域恢复证据。
## SITE-07 Monaco 历史 Unicode 数据检查点
三份原生成配方在冻结依赖下复现 RTL/Emoji 正则、imprecise token 表达式和全部
+3 -1
View File
@@ -2921,7 +2921,9 @@
"baselines/asr-combinations-validation.json",
"changes/2026-09-13-PKG-ASR-05-cors-combinations.md",
"baselines/asr-explicit-capture-validation.json",
"changes/2026-09-13-PKG-ASR-05-editor-generation.md"
"changes/2026-09-13-PKG-ASR-05-editor-generation.md",
"changes/2026-09-15-PKG-ASR-05-no-audio.md",
"baselines/asr-no-audio-validation.json"
]
},
{
+1
View File
@@ -41,6 +41,7 @@ export const installedTests = [
'asr-fallback.spec.js',
'asr-cors.spec.js',
'asr-capture-cors.spec.js',
'asr-no-audio.spec.js',
'asr-audio-track.spec.js',
'chromecast-lifecycle.spec.js',
'mediabunny-audio.spec.js',
+1
View File
@@ -20,6 +20,7 @@ test('Source invocation clears inherited artifact selection without changing cal
assert.notEqual(source.directory, installed.directory)
assert(installedTests.includes('ambilight-proxy.spec.js'))
assert(installedTests.includes('canvas-lifecycle.spec.js'))
assert(installedTests.includes('asr-no-audio.spec.js'))
for (const file of installedTests)
assert(fs.existsSync(path.resolve('test/browser', file)))
})
+137
View File
@@ -0,0 +1,137 @@
import fs from 'node:fs'
import process from 'node:process'
import { hash } from '../../refactor/scripts/releases.mjs'
import { browserCandidate } from '../helpers/browser-candidate.js'
import { expect, test } from './fixtures.js'
let candidate
test.beforeAll(async () => {
candidate = await browserCandidate('artplayer-plugin-asr', process.env.ARTPLAYER_ASR_ARTIFACT)
})
test.afterEach(async ({ page }, testInfo) => {
const observation = await page.evaluate(() => window.noAudioState?.()).catch(error => ({ error: error.message }))
await testInfo.attach('asr-no-audio-final', { contentType: 'application/json', body: JSON.stringify(observation ?? null) })
await page.evaluate(() => {
if (window.art && !window.art.isDestroy)
window.art.destroy()
})
})
for (const core of ['published', 'candidate']) {
for (const mode of ['default', 'capture']) {
test(`ASR ${mode} + ${core}: video-only input keeps playback and recovers audio capture after switching`, async ({ page }, testInfo) => {
const errors = []
page.on('console', (message) => {
if (message.type() === 'error')
errors.push(message.text())
})
await page.goto(`/test/player.html?core=${core}`)
const capabilities = await page.evaluate(() => ({ context: typeof (window.AudioContext || window.webkitAudioContext), worklet: typeof window.AudioWorkletNode, capture: typeof HTMLMediaElement.prototype.captureStream, mozCapture: typeof HTMLMediaElement.prototype.mozCaptureStream }))
await testInfo.attach('asr-no-audio-inputs', { contentType: 'application/json', body: JSON.stringify({ core, mode, capabilities, provenance: candidate.provenance, pluginSha256: hash(candidate.code), videoSha256: hash(fs.readFileSync(new URL('./media/pattern.mp4', import.meta.url))), audioSha256: hash(fs.readFileSync(new URL('./media/audio-tone.m4a', import.meta.url))), forcedException: false, physicalAudioOutput: false }) })
if (capabilities.context === 'undefined') {
expect(process.platform).toBe('win32')
expect(testInfo.project.name).toBe('webkit')
expect(capabilities.worklet).toBe('undefined')
test.skip(true, 'Windows WebKit has no WebAudio; not an Apple-device ASR acceptance')
}
expect(capabilities.worklet).toBe('function')
await page.addScriptTag({ content: candidate.code })
await page.evaluate((mode) => {
const Context = window.AudioContext || window.webkitAudioContext
const contexts = []
const tracks = []
const bindings = []
const chunks = []
window.AudioContext = class extends Context {
constructor(options) {
super(options)
contexts.push(this)
}
createMediaElementSource(element) {
bindings.push({ kind: 'direct', source: element.currentSrc })
return super.createMediaElementSource(element)
}
createMediaStreamSource(stream) {
bindings.push({ kind: 'stream', audio: stream.getAudioTracks().length, video: stream.getVideoTracks().length })
tracks.push(...stream.getTracks())
return super.createMediaStreamSource(stream)
}
}
window.art = new window.Artplayer({
container: '.player',
url: '/test/pattern.mp4',
volume: 0.6,
loop: true,
plugins: [window.artplayerPluginAsr({
...(mode === 'capture' ? { audioInput: { type: 'capture' } } : {}),
onAudioChunk({ pcm }) {
let peak = 0
for (const value of new Int16Array(pcm))
peak = Math.max(peak, Math.abs(value))
chunks.push({ source: window.art.video.currentSrc, peak, bytes: pcm.byteLength })
},
})],
})
document.querySelector('#play').onclick = () => window.art.play()
window.noAudioState = () => ({
bindings,
contexts: contexts.map(context => context.state),
tracks: tracks.map(track => ({ kind: track.kind, state: track.readyState })),
chunks,
media: { time: window.art.currentTime, source: window.art.video.currentSrc, paused: window.art.video.paused, error: window.art.video.error?.code || null, volume: window.art.volume, muted: window.art.muted },
})
window.noAudioPixels = () => {
const canvas = document.createElement('canvas')
canvas.width = 32
canvas.height = 18
const context = canvas.getContext('2d')
context.drawImage(window.art.video, 0, 0, 32, 18)
return [...context.getImageData(0, 0, 32, 18).data]
}
}, mode)
await expect.poll(() => page.evaluate(() => window.art.isReady)).toBe(true)
await page.locator('#play').click()
await expect.poll(() => page.evaluate(() => window.art.currentTime)).toBeGreaterThan(0.5)
const first = await page.evaluate(() => ({ pixels: window.noAudioPixels(), time: window.art.currentTime }))
await expect.poll(() => page.evaluate(() => window.art.currentTime)).toBeGreaterThan(first.time + 0.5)
expect(await page.evaluate(() => window.noAudioPixels())).not.toEqual(first.pixels)
const silent = await page.evaluate(() => window.noAudioState())
expect(silent.media).toMatchObject({ paused: false, error: null, volume: 0.6, muted: false })
expect(silent.chunks.every(chunk => chunk.peak === 0)).toBe(true)
if (mode === 'capture') {
expect(silent.bindings).toEqual([{ kind: 'stream', audio: 0, video: 1 }])
expect(silent.contexts).toEqual(['closed'])
expect(silent.tracks).toEqual([{ kind: 'video', state: 'ended' }])
expect(silent.chunks).toEqual([])
expect(errors).toHaveLength(1)
expect(errors[0]).toContain('[artplayerPluginAsr] Initialization failed:')
}
else {
expect(silent.bindings).toHaveLength(1)
expect(silent.bindings[0].kind).toBe('direct')
expect(silent.contexts).toEqual(['running'])
expect(silent.tracks).toEqual([])
expect(errors).toEqual([])
}
await testInfo.attach('asr-no-audio-playing', { contentType: 'application/json', body: JSON.stringify({ silent, errors: [...errors], changingDecodedPixels: true }) })
await page.evaluate(() => {
window.art.url = '/test/audio-tone.m4a?asr-no-audio=recovery'
})
await expect.poll(() => page.evaluate(() => window.art.video.currentSrc.includes('asr-no-audio=recovery') && window.art.video.readyState >= 2)).toBe(true)
await page.locator('#play').click()
await expect.poll(() => page.evaluate(() => window.noAudioState().chunks.filter(chunk => chunk.source.includes('asr-no-audio=recovery') && chunk.peak > 0).length)).toBeGreaterThanOrEqual(3)
const recovered = await page.evaluate(() => window.noAudioState())
expect(recovered.contexts).toEqual(mode === 'capture' ? ['closed', 'running'] : ['running'])
expect(recovered.bindings.filter(binding => binding.kind === 'direct')).toHaveLength(mode === 'capture' ? 0 : 1)
expect(recovered.media).toMatchObject({ paused: false, error: null, volume: 0.6, muted: false })
await testInfo.attach('asr-audio-recovered', { contentType: 'application/json', body: JSON.stringify(recovered) })
await page.evaluate(() => window.art.destroy())
await expect.poll(() => page.evaluate(() => window.noAudioState().contexts.every(state => state === 'closed'))).toBe(true)
expect(await page.evaluate(() => window.noAudioState().tracks.every(track => track.state === 'ended'))).toBe(true)
await expect(page.locator('.art-video-player')).toHaveCount(0)
})
}
}