mirror of
https://github.com/zhw2590582/ArtPlayer.git
synced 2026-10-29 02:51:20 -08:00
refactor(asr): [PKG-ASR-03] own typed audio capture and playback lifetimes
This commit is contained in:
1 parent
53c295819d
commit
0cf2af4840
33 files changed
+3492
-809
No files matched your search
@@ -1,33 +1,86 @@
|
||||
# ASR maintenance
|
||||
|
||||
The current implementation is still in `src/index.js`. TypeScript migration and
|
||||
resource ownership changes are tracked in the repository refactor plan as
|
||||
PKG-ASR-02 through PKG-ASR-06. Do not interpret the baseline tests as completion.
|
||||
The player captures first-channel audio and hands PCM/WAV to a caller-provided
|
||||
recognizer. There is no network request, microphone permission or embedded model.
|
||||
|
||||
- The default factory snapshots options. Registration creates the `asr` layer
|
||||
and subscribes to player play, pause, volumechange and destroy events.
|
||||
- Play creates an AudioContext and tries a media element source, then a captured
|
||||
MediaStream source. An inline Blob contains the `recorder-processor` Worklet.
|
||||
- Worklet messages supply first-channel Float32 samples. The interval callback
|
||||
encodes little-endian mono PCM16 and a 44-byte-header WAV and invokes the
|
||||
caller's `onAudioChunk({ pcm, wav })`. A returned string replaces subtitles.
|
||||
- `append` splits punctuation and sets HTML in `div.art-asr-line`; `hide` changes
|
||||
display only. `src/style.less` owns layout and existing player CSS variables.
|
||||
- `stop()` is actually asynchronous and releases audio resources, but retains
|
||||
playback subscriptions. It is not a permanent disable API. There is no public
|
||||
`start()` method. Public declarations currently describe stop as returning void.
|
||||
| File | Responsibility |
|
||||
| --- | --- |
|
||||
| src/index.ts | Snapshot options, register player events, return the compatible facade and install the stylesheet |
|
||||
| src/types.ts | Internal options, result and browser capability types; public declarations remain in types/ |
|
||||
| src/subtitles.ts | Punctuation/HTML rendering and the owned auto-hide timer |
|
||||
| src/capture.ts | Initialization deduplication, active generation, FIFO consumption and one pending recognizer call |
|
||||
| src/audio-graph.ts | AudioContext, element/stream source, Worklet loading, playback routing and recorder ownership |
|
||||
| src/sample-queue.ts | Copied sample blocks with a linked queue and head offset; preserves partial blocks and tails |
|
||||
| src/encoding.ts | Pure PCM16LE/WAV encoding, including historical clamping and the 44-byte mono WAV header |
|
||||
| src/worklet.ts | Inline processor code; first-channel messages and the existing recorder-processor registration name |
|
||||
| src/style.less | Existing art-layer-asr layout, art-asr-line content and player CSS variables |
|
||||
|
||||
This package does not implement recognition or open an ASR network connection.
|
||||
The demo service and its WebSocket belong to the caller. Do not run that service
|
||||
to validate audio capture; use local media and a local callback.
|
||||
Dependencies flow from the facade into subtitles/capture, then from capture into
|
||||
the graph, queue and pure encoder. Audio modules do not receive the whole player.
|
||||
The Worklet is loaded from a temporary Blob before binding the media element, so
|
||||
a failed module load can close its context without taking over video playback.
|
||||
|
||||
The initial implementation has suspected chunk loss, partial initialization
|
||||
cleanup and stale callback issues. Preserve the frozen historical tests while
|
||||
adding candidate regression tests when fixing these. Keep callback/stop typing
|
||||
compatible with historical assignments, and preserve public entrypoints, HTML
|
||||
rendering behavior, layer names and stylesheet identity.
|
||||
## States and ownership
|
||||
|
||||
From the repository root, `yarn test:asr` verifies historical public behavior;
|
||||
`yarn test:baseline` includes archive provenance. `yarn dev artplayer-plugin-asr`
|
||||
and `yarn build artplayer-plugin-asr` use repository build tools. Real browser
|
||||
audio and lifecycle validation remains a separate required migration step.
|
||||
Play shares one pending initialization. Successful setup attaches one recorder
|
||||
and starts one interval. Each tick takes exactly one complete block; an incomplete
|
||||
block stays queued. Only one callback is pending within the active generation.
|
||||
Rejections are logged and later ticks continue. Pause, source restart, stop and
|
||||
destroy invalidate old recognizer completions and discard obsolete queued audio.
|
||||
Each recorder also checks its own identity, rejecting messages from retired ports.
|
||||
|
||||
Pause detaches the recorder and cancels the interval. The direct media source
|
||||
continues through its playback gain to the destination. A source switch reuses the
|
||||
direct source and replaces the recorder; a captured-stream fallback is closed and
|
||||
reacquired because captured tracks can end on source changes.
|
||||
|
||||
`stop()` returns `Promise<void>` and releases ASR work. An already displayed
|
||||
subtitle retains its original auto-hide deadline; obsolete callbacks cannot add timers.
|
||||
It retains playback subscriptions and allows explicit append or a later play.
|
||||
For a directly bound video, the player keeps its original source/context and audio
|
||||
route until `art.destroy()`. Closing that context at every stop caused Chromium
|
||||
to lose audio on restart; a media element cannot simply bind a replacement source.
|
||||
For a captured fallback, stop closes its disposable context and stops its tracks.
|
||||
Before direct binding, failed/cancelled initialization closes its context and
|
||||
revokes the Worklet URL. Destroy closes all contexts, disconnects nodes, stops
|
||||
captured tracks, cancels subtitle timers and removes subscriptions.
|
||||
|
||||
This lifetime follows the [Web Audio close semantics](https://www.w3.org/TR/webaudio-1.0/#dom-audiocontext-close)
|
||||
and [media-element source behavior](https://www.w3.org/TR/webaudio-1.0/#MediaElementAudioSourceNode).
|
||||
Captured-stream source changes follow [Media Capture from DOM Elements](https://www.w3.org/TR/mediacapture-fromelement/#html-media-element-media-capture-extensions).
|
||||
|
||||
Queued audio is limited to the larger of 60 seconds or two configured chunks. A caller that
|
||||
cannot keep up gets an explicit console error and capture pauses, leaving normal
|
||||
playback routed. This avoids an unbounded backlog; it does not claim lossless
|
||||
recognition under overload. A later play can restart capture. The caller owns
|
||||
cancellation of its own network requests; obsolete completions cannot repaint.
|
||||
|
||||
## Compatibility and verification
|
||||
|
||||
The factory still snapshots options and registers synchronously, returning
|
||||
`name`, `stop`, `hide`, `append`. There is no public start method. Append preserves
|
||||
HTML, punctuation splitting, replacement, ignored non-string values and the
|
||||
historical `length=0` behavior. The CJS function has a non-enumerable self `default`
|
||||
alias for consumers of the 2.0 namespace, while preserving the callable 2.1 form.
|
||||
|
||||
Root public declarations have not yet migrated: they describe stop as void and
|
||||
the callback as void/Promise<void>. PKG-ASR-04 owns exact asynchronous public types,
|
||||
old assignment consumers, module forms and installed tarball acceptance. Internal
|
||||
strict TypeScript does not prove that public type migration complete.
|
||||
|
||||
From the repository root:
|
||||
|
||||
- `yarn test:asr`: frozen public/audio observations, queue/encoding parity and
|
||||
candidate lifecycle regressions. Historical defects remain separate tests.
|
||||
- `yarn test:browser test/browser/asr-audio.spec.js`: local AAC, actual Worklet,
|
||||
published/current core, pause/resume, source switch, stop/restart and destroy.
|
||||
Set `ARTPLAYER_ASR_ARTIFACT` to an actual main/legacy file to test that artifact.
|
||||
- `yarn typecheck`: includes the package strict/noUncheckedIndexedAccess project.
|
||||
- `yarn dev artplayer-plugin-asr` and `yarn build artplayer-plugin-asr`: normal
|
||||
repository development and three-format production builds.
|
||||
|
||||
Windows Playwright WebKit 26.6 lacks AudioContext and AudioWorkletNode. Its native
|
||||
ASR cases remain explicit skips and do not establish Safari/device compatibility.
|
||||
Real fallback track replacement, native volume/mute, CORS, unsupported media and
|
||||
physical devices remain PKG-ASR-05 acceptance work. Do not use the demo's external
|
||||
recognizer service as a prerequisite for local audio verification.
|
||||
Reference in new issue
Block a user