mirror of
https://github.com/4gray/iptvnator.git
synced 2026-10-08 17:06:15 -08:00
feat(downloads): download completed Xtream catch-up programmes as TS (#1572)
* feat(epg): copy catch-up programme URLs without changing playback * feat(downloads): save completed Xtream archive programmes as TS * fix(epg): let newer archive copy requests supersede pending work * fix(downloads): protect archive partials and independent submissions * fix(downloads): verify archive identity through finalization * fix(downloads): bound archive storage and capture cleanup entries * fix(downloads): preserve archive ownership across failure paths * fix(downloads): recover explicitly verified archive completions * fix(downloads): journal archive promotion before publishing files * fix(downloads): reset archive proof before an explicit restart * fix(downloads): preserve archive recovery ownership and interruption * fix(downloads): verify durable archive identity at resume open * fix(downloads): fence archive commands during completion commit * fix(downloads): persist archive ownership throughout its lifecycle * fix(downloads): protect archive removal and missing-file recovery * fix(downloads): journal private cleanup captures for recovery * fix(downloads): journal active archive cleanup before removal * fix(downloads): clean settled archives before deleting stale rows * fix(downloads): preserve archive ownership on removal and resubmission * fix(downloads): recover proven archive completions before retry * fix(downloads): recover local archives before remote transfer checks * test(downloads): resolve archive fixture from workspace root * fix(downloads): distinguish reused archive inodes by creation time * fix(downloads): bind fresh archive reservations to owned files * fix(downloads): clean reservations when ownership writes fail * fix(downloads): commit archive reservation and ownership atomically * fix(downloads): retain captures until replacement restoration succeeds * fix(downloads): require durable ownership before cleanup relocation * fix(downloads): preserve 64-bit archive file identities on Windows * refactor(release): keep capture fixture constants in their shared module * fix(downloads): preserve the last link of captured foreign files * fix(downloads): expose retained archive recovery files * fix(downloads): keep recovery instructions open while copying
This commit is contained in:
1 parent
a7f3860102
commit
bad8a0991e
117 files changed
+7510
-880
No files matched your search
@@ -7,10 +7,173 @@ views. Backend work is handled in the Electron process while the Angular
|
||||
renderer exposes the global `/workspace/downloads` page, source-scoped route
|
||||
variants, contextual buttons, and theme-aware styling.
|
||||
|
||||
## Xtream archive downloads
|
||||
|
||||
The desktop Xtream Live TV programme dialog offers **Download programme (TS)**
|
||||
for completed, available catch-up programmes in both timeline and list views.
|
||||
The host captures the playlist, channel and original programme timestamps before
|
||||
resolving the existing timeshift URL (including the panel timezone). A channel
|
||||
change while the dialog is open invalidates its action. Downloading does not
|
||||
select the programme or change playback. Collection views retain archive URL
|
||||
copying; archive downloads are initially exposed through Xtream Live TV only.
|
||||
M3U, Stalker and PWA archive downloads, HLS assembly and recording future broadcasts
|
||||
are outside this feature.
|
||||
|
||||
`EpgArchiveDownloadService` submits the resolved URL and the playlist's playback
|
||||
header allowlist to the existing desktop queue. Pending submission suppression
|
||||
is per programme identity, so slow resolution does not discard a different
|
||||
programme's request. HLS URLs are refused explicitly;
|
||||
the backend also checks the response type and every 188-byte MPEG-TS packet.
|
||||
No transcoder or media helper is required. The file always uses `.ts`.
|
||||
|
||||
The `catchup` download type stores `programme_start` and JSON metadata containing
|
||||
channel name, start/stop Unix seconds and the known archive expiry. Its unique
|
||||
identity is `(playlist_id, xtream_id, programme_start)`; movies and episodes keep
|
||||
their existing identity. `download-schema.ts` widens the SQLite CHECK in a
|
||||
transaction after older additive migrations, preserving row IDs, files, headers,
|
||||
resume state and metadata. It also replaces the former global unique index with
|
||||
separate catch-up and non-catch-up indexes.
|
||||
|
||||
Only ended programmes can be enqueued. Known expiry is checked on enqueue,
|
||||
retry, resume, missing-file recovery and when the queued transfer starts.
|
||||
Pause keeps its owned partial file, but **resume/retry restarts from byte zero**,
|
||||
without Range, If-Range, or automatic reconnect append. The queue explains this.
|
||||
Before truncating, `download-catchup-output.ts` rejects symlinks and hardlinks,
|
||||
opens without truncation (with O_NOFOLLOW where available), checks descriptor
|
||||
identity against lstat, then truncates and writes through that same descriptor.
|
||||
An absent partial is created exclusively, so a replaced path cannot redirect writes.
|
||||
The task retains that descriptor's device/inode identity through promotion:
|
||||
`download-catchup-finalize.ts` verifies the source and published file, rejects a
|
||||
replaced partial, and copies from the verified descriptor on filesystems without
|
||||
hardlinks. Existing destination files are never overwritten. Cleanup atomically
|
||||
moves the public pathname into a private temporary directory before checking its
|
||||
identity and removing it. A captured replacement is restored with no-clobber
|
||||
linking; if restoration is unavailable or the original pathname is occupied,
|
||||
the file is retained in `.iptvnator-cleanup-*/entry` and its recovery location
|
||||
is logged. Cleanup never recursively deletes a nonempty quarantine. This closes
|
||||
the predictable-path check/unlink window; it does not isolate files from
|
||||
same-user processes that deliberately enter the private temporary directory.
|
||||
Active failure and cancellation use the same captured transfer identity; a
|
||||
partial that was never safely opened is preserved instead of being deleted.
|
||||
`download_archive_finalizations` is a write-ahead SQLite journal keyed by download
|
||||
ID (cascade-deleted with the download). It records the path, size, source identity
|
||||
and expected final identity **before hardlink promotion**, or before the first
|
||||
byte is written to an exclusively created copy destination. Thus even a completed
|
||||
unknown-length file has durable proof before the completion-status write. Startup
|
||||
requires that proof and a matching regular file, identity and size to recover an
|
||||
archive; termination before promotion leaves the verified partial paused. An
|
||||
owned incomplete copy is removed by journal identity before the source resumes. The
|
||||
journal also makes startup partial cleanup identity-aware. A journaled partial
|
||||
whose identity changed is preserved and detached from the failed row; Retry
|
||||
reserves a fresh path instead of adopting or truncating it. The journal remains with
|
||||
completed archives until they are removed. An explicitly restarted transfer
|
||||
carries the journaled source identity to output opening and checks
|
||||
it against the opened descriptor. Only then does it replace the previous proof,
|
||||
before truncation or writes, replacing the old finalization proof with a
|
||||
transfer-phase journal containing the newly opened identity. That identity
|
||||
survives pause, failure and restart without claiming the file is complete. A
|
||||
rejected replacement is detached from the failed row without deletion, so Retry
|
||||
reserves a new path. Retained catch-up paths without any durable proof are also
|
||||
preserved and redirected to a fresh reservation. On a final-path collision,
|
||||
archives never relocate a retained partial: only a verified owned partial is
|
||||
discarded before reserving a new filename, because archive retries restart at zero. The transfer proof is upgraded to finalization proof only
|
||||
after validated EOF. Process-local proof allows immediate
|
||||
recovery after a transient completion DB error without waiting for a restart.
|
||||
Fallback copying observes pause/cancel between bounded 64 KiB reads/writes and
|
||||
before publication completes. Interruption removes only the owned copy and
|
||||
leaves the source for the runtime pause/cancel handler. Once publication identity
|
||||
and size pass the final check, the task synchronously enters completion commit
|
||||
before awaited partial cleanup and the SQLite completion write. Pause/cancel then
|
||||
return false without setting flags; a command is never accepted and subsequently
|
||||
overwritten by completion. Remove rejects this committing row before partial
|
||||
cleanup or deletion, and Clear completed skips it, preserving the cascading
|
||||
journal until completion finishes. Remove waits for an accepted active archive
|
||||
cancellation to settle before reading/deleting its row; queued archives are
|
||||
canceled first, and a concurrent new runtime attempt blocks removal. A settled
|
||||
archive still marked downloading after a failed status write also retries
|
||||
journal cleanup before row deletion. Remove/Clear preserve a final file whose
|
||||
journaled device/inode/creation-time identity and full size prove completed promotion, even when completion
|
||||
status writes failed and its stored status is stale. Repeat submissions, Retry
|
||||
and Resume restore such a journal-proven completion in place before any new
|
||||
transfer or ownership reset, before expiry, provider DNS and new-folder checks
|
||||
that apply only to another remote transfer; retained cleanup failures keep their
|
||||
journal. Remove, Clear completed and missing-file
|
||||
re-download and repeated programme submissions use journal-backed private
|
||||
capture for archive partial cleanup;
|
||||
unknown or replaced entries are preserved. Ownership reads device/inode as BigInt and journals decimal strings without
|
||||
losing 64-bit Windows file references, alongside a positive file creation timestamp, so inode reuse after unlink cannot
|
||||
bless a new entry; proofs lacking creation time remain untrusted. Fresh
|
||||
reservations capture this identity from their exclusive creation descriptor and
|
||||
commit it together with the downloads row path/name in one SQLite transaction
|
||||
before the HTTP wait. A committed reservation is therefore recoverable even if
|
||||
the process exits before transfer setup. Existing partials without matching expected
|
||||
ownership are never truncated, including before the first response. Before capture, a synchronous SQLite
|
||||
write records `partialCleanupPath` (or `finalCleanupPath` for failed promotion)
|
||||
in the existing ownership proof. Active cancellation/failure, promotion and
|
||||
startup recovery use the same journal-backed cleanup as manual actions. Cancel,
|
||||
failure and removal of an unfinished attempt also clean its journaled final target;
|
||||
removing a completed row preserves its media. Retry cleans an incomplete owned
|
||||
final before reserving a destination. A failed
|
||||
unlink or replacement restoration keeps that durable pointer and blocks journal
|
||||
deletion/replacement. Later cleanup retries no-clobber restoration of captured
|
||||
foreign entries; an occupied public path or unsupported hardlinks preserves both
|
||||
the capture and its journal for recovery. Even after a foreign entry is restored,
|
||||
its private recovery copy is never automatically unlinked: another process could
|
||||
remove the public link first. Remove/Clear return a structured recovery path and open a persistent, localized
|
||||
dialog with the full path, Copy recovery path and manual recovery instructions.
|
||||
Copy keeps the dialog open, including on clipboard failure; Close dismisses it.
|
||||
After the user recovers it and explicitly removes that private copy, cleanup may release the
|
||||
journal. Ordinary owned-file cleanup remains automatic. Remove/Clear, Retry/Resume and fresh
|
||||
reservations retry identity-verified cleanup, including after restart and on
|
||||
filesystems without hardlinks. Cleanup remains synchronous after the
|
||||
runtime guard, so a completion transition cannot interleave with unlink.
|
||||
Missing archive re-downloads claim a fresh reservation instead of inheriting the
|
||||
completed file's identity.
|
||||
If the initial reservation journal write fails, persistence is retried once
|
||||
before journal-backed cleanup. If SQLite remains unavailable, the empty
|
||||
reservation stays at its public path; no unjournaled entry is relocated. An
|
||||
empty fallback-copy target whose final proof cannot commit is likewise preserved.
|
||||
A kill between exclusive reservation/copy-file creation and its identity journal
|
||||
commit, or persistent SQLite failure before that commit, can leave an unowned
|
||||
**empty** destination: no bytes are written before the commit.
|
||||
Recovery preserves that file rather than guessing ownership; Retry uses a
|
||||
numbered free destination. SQLite and filesystem creation cannot commit
|
||||
atomically, and portable rename cannot guarantee no-clobber publication on the
|
||||
filesystems that need this fallback. This bounded orphan is preferred to deleting
|
||||
or overwriting an unrelated file.
|
||||
Cancellation of queued/paused archives uses the same journal-backed cleanup as
|
||||
Remove; unproven or replaced entries, including symlinks, are preserved.
|
||||
A retained archive cannot use the VOD byte-count completion shortcut. Transfers
|
||||
have a 30-second idle timeout and a total deadline of twice programme duration
|
||||
plus ten minutes, capped at 24 hours. Transfers also stop at the smallest of a
|
||||
100 Mbit/s budget for the programme duration plus one minute, 64 GiB, and
|
||||
half of the initial available space after subtracting a 1 GiB reserve (reserving a second
|
||||
copy for filesystems without hardlinks). The retained partial is safely truncated
|
||||
through its verified descriptor before computing this budget, so Resume/Retry
|
||||
can reuse its released space. Known Content-Length values above
|
||||
that budget are rejected before writing; unknown-length responses are counted
|
||||
before forwarding chunks. Free space is rechecked every 16 MiB to account for
|
||||
other disk activity, and copy headroom is checked again against the final byte
|
||||
count after the output stream closes, including short tails below that interval. These safety limits apply to TS archives only. Failure never promotes the partial to the
|
||||
library, and errors omit credential-bearing URLs.
|
||||
|
||||
Completion means clean HTTP EOF, matching Content-Length when supplied, and
|
||||
valid nonempty TS packet framing. **It does not verify media duration or the
|
||||
provider's programme boundaries.** A provider can return a shorter valid clip
|
||||
with clean EOF; this version cannot identify that without demuxing duration.
|
||||
Unknown-length responses show indeterminate progress until EOF.
|
||||
|
||||
Completed archive cards appear under All with programme title, channel and
|
||||
broadcast date. Clicking a card plays the local file through the download action;
|
||||
it does not navigate to a movie or series detail route. Files and metadata remain
|
||||
available after restart and after the source archive expires.
|
||||
|
||||
## Backend responsibilities
|
||||
|
||||
- **Queue control (`apps/electron-backend/src/app/events/database/download-runtime.ts`)**
|
||||
`DownloadTask` mirrors a row of the shared `downloads` table (type `Download` in `libs/shared/database/src/lib/schema.ts`) plus transient cancel/pause/progress helpers (shared task types live in `download-task.ts`). Request validation and row creation live in `download-requests.ts`, while `downloads.events.ts` stays focused on IPC registration. `enqueueDownload()` pushes the task onto `downloadQueue` and triggers `processQueue()`. `processQueue()` keeps one active download, updates the row to `downloading`, and calls `startDownload()`. The byte transfer itself lives in `download-transfer.ts`, finalization and retained-partial persistence in `download-finalize.ts`, and the renderer update broadcast in `download-broadcast.ts`.
|
||||
`DownloadTask` mirrors a row of the shared `downloads` table (type `Download` in `libs/shared/database/src/lib/schema.ts`) plus transient cancel/pause/progress helpers (shared task types live in `download-task.ts`). Request validation and row creation live in `download-requests.ts`, retry/resume flows in `download-resume-requests.ts`, removal/terminal cleanup in
|
||||
`download-removal-requests.ts`, while `downloads.events.ts` stays focused on IPC registration. `enqueueDownload()` pushes the task onto `downloadQueue` and triggers `processQueue()`. `processQueue()` keeps one active download, updates the row to `downloading`, and calls `startDownload()`. The byte transfer itself lives in `download-transfer.ts`, finalization and retained-partial persistence in `download-finalize.ts` (ordinary file promotion in `download-file-finalize.ts`, archive completion in
|
||||
`download-catchup-completion.ts`, proven completion recovery in `download-catchup-recover-completion.ts`, target reservation in `download-runtime-reservation.ts`), cancellation/pause persistence in `download-runtime-persistence.ts`, and the renderer update broadcast in `download-broadcast.ts`.
|
||||
- **Range-aware transfer (`download-transfer.ts`)**
|
||||
The transfer streams the response through the backend's validated Axios redirect helper instead of `electron-dl`, and always requests `Accept-Encoding: identity`: Range offsets, totals, and the persisted `.part` must describe the same representation, and Axios's transparent gzip/brotli decoding would put decoded bytes on disk while every counter speaks encoded bytes. Headers (user agent, referer, origin) are persisted in `request_headers` and re-applied through the same allowlist when read back on retry/resume. Fresh Xtream movie and series-episode downloads propagate the playlist's configured headers, using its User-Agent when present and otherwise sharing the provider-compatible `XTREAM_CLIENT_USER_AGENT` used by Xtream API requests and stream probes. Retry, resume, and missing-file recovery resolve the owning playlist type and add that fallback to legacy Xtream rows without a stored User-Agent; known Stalker rows are left unchanged. Download rows deliberately outlive individually deleted playlists, so a headerless legacy row whose source no longer exists receives the same IPTV-player fallback because its original provider type cannot be recovered. Active pause/cancel operations abort the current request with `AbortController`; pause keeps the partial file and cancel removes it. Resume checks the existing `.part` size (rejecting anything that is not a regular file, so a symlink planted while paused is never followed). The first response's strong `ETag` (or `Last-Modified`) is persisted in `resume_validator`; a partial carrying that validator resumes with `Range: bytes=<offset>-` plus `If-Range`, so the server itself proves the entity is unchanged. A retained partial **without** a validator resumes through overlap verification instead (`download-overlap.ts`): the `Range` request rewinds by up to 256 KiB (`OVERLAP_VERIFICATION_BYTES`) and a transform stream compares that replayed window byte-for-byte against the partial's tail before anything is appended — a self-made validator for the many Xtream panels that send neither header. A mismatching overlap truncates the `.part` and restarts the transfer from byte zero (`OverlapMismatchError`); a partial smaller than the overlap window is verified in full from byte zero over a plain request and appended to — never rewritten in place, so a reconnect that dies early can only grow the file. Success requires the verifier to have consumed its ENTIRE window: a response that ends inside the overlap is an ordinary retained interruption when the stream died early, but a response that delivered its complete AUTHORITATIVE total inside the window — whether it then closed cleanly or reset — proves the remote entity shrank and restarts from scratch; the old suffix is never finalized as a completed file. An HTTP 416 answer to a resume request is classified by `classifyRangeNotSatisfiable()`: it COMPLETES only an exact-EOF request with identity proof (`If-Range`-backed, or the EOF probe that follows a fully verified overlap replay) whose stated `bytes */N` equals the partial — a bare length match on a rewound request proves nothing about whose bytes are on disk; it RESTARTS only when a STATED total proves the entity shrank — the total sits below a rewound request's first byte, or at it (the rewound range beginning exactly at the new EOF), or below the partial at an exact-EOF request; every length-less, ambiguous, or contradictory 416 RETAINS the partial, and none of these paths ever reaches generic cleanup. A validator promoted by a complete overlap match survives mid-append failures too: the promotion also runs on the error path, and retained-failure and pause persistence write `resume_validator` from the task, so later attempts resume via `If-Range` instead of replaying the window — without this, a server whose per-connection cap barely exceeds the window would stall out on sub-threshold progress. A verify-append attempt promotes the response's `ETag`/`Last-Modified` onto the row only after the complete overlap matched; until then the retained bytes are unproven and blessing them with a validator would let the next resume `If-Range`-append onto a foreign prefix. The response's TOTAL stays equally uncommitted (task and row) until the overlap matched — a persisted total equal to the unverified partial's size would let the completed-partial shortcut finalize unproven bytes after a pause, crash, or retained failure. Retained-interruption persistence keeps the live task in sync with the row (a stale falsified total would make the next reconnect's resume-offset guard reject the partial). Overlap replay re-counts bytes from the rewound offset, so reported progress is floored at the partial's retained size whenever the transfer appends — a response that ends inside the overlap can never move displayed progress backwards. A `206 Partial Content` answer must start at the requested offset (`Content-Range` is verified) before bytes are appended; any other 2xx answer Line truncated
|
||||
- **Destination collision policy**
|
||||
@@ -294,12 +457,12 @@ display snapshot via `playlistDisplayLabel`).
|
||||
- **Lifecycle tracking** is owned by `EmbeddedMpvRecordingTracker`
|
||||
(`apps/electron-backend/src/app/services/embedded-mpv-recording-tracker.ts`):
|
||||
explicit start/stop hooks in `EmbeddedMpvNativeService` plus a
|
||||
session-snapshot observer. The stop hook is a *request*, not an outcome —
|
||||
session-snapshot observer. The stop hook is a _request_, not an outcome —
|
||||
`addon.stopRecording()` only dispatches (async mpv property set, or a
|
||||
command written to the frame-copy helper), so finalization always waits for
|
||||
the snapshot reporting the recording inactive; statting or unlinking
|
||||
earlier would report a short recording as failed and could delete bytes mpv
|
||||
is still flushing. macOS native-view clears `recordingActive` *before*
|
||||
is still flushing. macOS native-view clears `recordingActive` _before_
|
||||
dispatching the async property set and restores it if that request is
|
||||
rejected, so an inactive snapshot must additionally survive a 1.5 s settle
|
||||
window (three poll cycles) before it counts as an acknowledgement; a revived
|
||||
@@ -355,7 +518,7 @@ display snapshot via `playlistDisplayLabel`).
|
||||
`RecordingStartMetadata` (channel name/logo, playlist id + display-label
|
||||
snapshot, source type, EPG key, current program) that flows
|
||||
`WebPlayerViewComponent → EmbeddedMpvPlayerComponent →
|
||||
EmbeddedMpvControlsAdapter → EmbeddedMpvRecordingStartOptions.metadata`.
|
||||
EmbeddedMpvControlsAdapter → EmbeddedMpvRecordingStartOptions.metadata`.
|
||||
`EmbeddedMpvPlayerComponent` watches the session snapshot for the
|
||||
active→inactive recording edge and emits `recordingStopped` — one owner for
|
||||
every trigger, including a Stop clicked in the download manager, which
|
||||
@@ -377,14 +540,14 @@ display snapshot via `playlistDisplayLabel`).
|
||||
target path — that is how a recording spanning a program boundary lists
|
||||
every covered show. The handler is deliberately independent of finalization: it
|
||||
looks the row up in **any** status (the newest for that path — `openSync
|
||||
('wx')` keeps a reserved path exclusive while its recording owns it) and
|
||||
('wx')` keeps a reserved path exclusive while its recording owns it) and
|
||||
`finalize()` never touches `programs_json`, so the two writes commit in
|
||||
either order. The only wait is the tracker's queue drain, which guarantees
|
||||
the row's INSERT exists — no deadline, and therefore no way for a one-shot
|
||||
enrichment to be dropped by a clock. A recording stopped while no player is mounted on that
|
||||
channel keeps its start snapshot.
|
||||
- **IPC surface** (`recordings.events.ts`): `RECORDINGS_GET_LIST/GET/STOP/
|
||||
REMOVE/UPDATE_PROGRAMS/REVEAL_FILE/PLAY_FILE` plus the dedicated
|
||||
REMOVE/UPDATE_PROGRAMS/REVEAL_FILE/PLAY_FILE` plus the dedicated
|
||||
`RECORDINGS_UPDATE_EVENT` bare ping (not shared with downloads, so
|
||||
recording transitions do not force availability-probed download refetches).
|
||||
Active rows are decorated with a live `fs.stat` size — `file_size_bytes` is
|
||||
|
||||
@@ -316,3 +316,11 @@ or without, the account-info check. The same timezone lets
|
||||
clock the panel wrote them in (`parseXtreamServerLocalDateTime`);
|
||||
`start_timestamp` still wins whenever it is present. Formatting uses
|
||||
`hourCycle: 'h23'`, so server midnight renders as `00`, never `24`.
|
||||
|
||||
### Downloading completed catch-up programmes
|
||||
|
||||
Desktop Xtream Live TV exposes a TS-only download action in programme details.
|
||||
It reuses the canonical timeshift resolver and original timestamps, preserves
|
||||
playback headers and does not change playback. See
|
||||
[Download Manager](download-manager.md#xtream-archive-downloads) for identity,
|
||||
restart, expiry and transport-completion limits.
|
||||
Reference in new issue
Block a user