iptvnator/docs/architecture/host-connectivity-guard.md

# Host Connectivity Guard

Per-host circuit breaker for portal requests, in both processes that make them:
the Electron main process and the self-hosted web backend.

## The problem

Every request to an unreachable portal costs its full axios timeout — 30 s for
`XTREAM_REQUEST`, 15 s for `STALKER_REQUEST` (30 s for `create_link`), and the
same budgets on the web backend's `/xtream` and `/stalker` routes. Browsing a
dead portal's catalog issues dozens of those back to back, which shows up as
30-second spinners and a log full of identical failures. Once a host has refused
to answer twice in a row there is nothing left to learn from waiting again.

The web backend had a worse version of the same problem first: its proxy routes
passed no `timeout` at all, so a provider that accepted a connection and then
went silent held the request until the OS gave up on the TCP connection. A
breaker is only useful once _not answering_ is bounded, which is why the timeouts
and the breaker landed there together.

## Where it lives

`libs/shared/host-health` (`@iptvnator/shared/host-health`, tagged
`scope:shared` / `domain:shared-runtime` / `type:util`) — the breaker class, the
failure classification and the redirect-attribution helpers, with no transport,
logger or process singleton of its own. The owning app supplies the clock and
decides how many guards exist.

Each runtime owns its instance:

- **Electron** — `apps/electron-backend/src/app/util/host-connectivity-guard.ts`
  holds one guard for the whole process, wired into both IPC handlers. The
  handlers are the choke point that sees _all_ traffic to a host. The renderer's
  `executeStalkerRequest` is not: it has four documented bypasses (auth,
  endpoint discovery, account info, the row-less stream resolver), and that is
  exactly the traffic that hits dead hosts.
- **PWA** — `apps/web-backend/src/app/host-guard.ts` guards the `/xtream` and
  `/stalker` proxy routes. The guard instance is injected through
  `WebBackendAppOptions.hostGuard`, alongside `now` and `guid`, so specs drive
  it with a clock they own.

### Request timeouts are the precondition

A breaker keyed on "the endpoint did not answer" is only useful once _not
answering_ is bounded. `apps/web-backend` originally passed no `timeout` at all,
so a silent host hung on OS-level TCP timeouts. Both runtimes now use the same
budgets: Xtream 30 s, Stalker 15 s (30 s for `create_link`, which mints a stream
URL before answering), playlist and XMLTV downloads 30 s.

Those numbers are safe for large downloads. On axios 1.20's native HTTP
transport (`maxRedirects: 0`) each hop's `timeout` is **not** a wall-clock deadline for the
whole response: it bounds the time to response headers and then continues as the
socket's inactivity timeout for the body. A multi-megabyte XMLTV file that keeps
delivering bytes is never cut off mid-transfer — only a stalled one is. The web
backend explicitly restores socket inactivity handling while reading the final
stream response, because axios stream mode stops its timeout handler at headers.
Truncated, malformed and timed-out final bodies retain their received response
status for reachability classification; cancellation retains its own semantics.

### Scope: portal calls only

Both runtimes guard the portal API paths and nothing else. Playlist and XMLTV
downloads (`/parse`, `/parse-xml`, and their Electron equivalents) get the
timeouts but not the breaker, deliberately:

- The problem being solved is a catalog fan-out — dozens of requests to one
  endpoint back to back. A download is a single request.
- A download is usually the direct result of the user asking for it (the
  add-playlist dialog sends `PLAYLIST_PARSE_BY_URL`). Refusing an immediate
  retry is a regression, not a protection, and there is no natural reset site on
  that path the way portal Retry has one.

## Desktop preference and account feedback

Settings > General > Portal connections offers **Pause requests to unavailable
portals**, enabled by default. `Settings.portalConnectivityGuard` is default-on
for missing or non-boolean legacy values; only explicit `false` opts out. The
form stages edits until Save and persists the choice through the normal settings
store. `SETTINGS_UPDATE` mirrors it to Electron's `PORTAL_CONNECTIVITY_GUARD`
config key and applies it immediately. `SettingsEvents.bootstrapSettingsEvents`
loads that mirror before the renderer is loaded, so a saved opt-out already
applies to the first portal request after restarting.

The preference controls both Xtream and Stalker. A real preference transition
replaces the Electron guard and its weak set of admitted request tokens, clearing
existing cooldowns and ignoring completions from the old generation. Disabled
requests return no token and cannot contribute failures when the user re-enables
the guard. Saving an unchanged value preserves existing evidence.
`IPTVNATOR_DISABLE_CONNECTIVITY_GUARD=1` (or `true`) still overrides an enabled
preference. The UI is capability-gated by `supportsPortalConnectivityGuard`
(`updateSettings` plus `resetHostConnectivityGuard`); the PWA has no client-side
switch for its shared server-wide guard.

Both account-info dialogs classify the existing cross-process guard message and
show a localized **Requests temporarily paused** explanation with **Retry now**.
Stalker keeps cached account data visible and offers the same retry beside the
paused refresh notice. Retry resets before requesting again; actual network
failures retain the generic unavailable state. These notices describe the last
request outcome, not a live countdown. Same-millisecond sibling counting is
handled by monotonic admission ids (#1438).

## Rules

Being wrong here means refusing to talk to a portal that works, so every rule
errs towards contacting the host:

|             |                                                                        |
| ----------- | ---------------------------------------------------------------------- |
| Trip        | 2 consecutive host-level failures within an inclusive 120 s window     |
| Open for    | 30 s (`OPEN_DURATION_MS`), matching the repo's other cooldowns         |
| Half-open   | exactly ONE trial request; the rest keep fast-failing until it settles |
| Reset       | any HTTP response — 200, 404, even 502; an accepted TCP connection clears the streak but not an open breaker |
| Key         | `URL.origin` — scheme, host **and** port (see below)                   |
| Kill switch | `IPTVNATOR_DISABLE_CONNECTIVITY_GUARD=1` (read per call)               |

**Host-level failure** means an error with no HTTP response whose code is one of
`ETIMEDOUT`, `ECONNABORTED`, `ENOTFOUND`, `EAI_AGAIN`, `ECONNREFUSED`,
`EHOSTUNREACH`, `ENETUNREACH` — observed **before the TCP connection was
established**. `ECONNRESET` is deliberately excluded: a reset mid-transfer
happens on hosts that are very much alive. Cancelled requests (`ERR_CANCELED`)
and SSRF-policy refusals are `inconclusive` — they say nothing about
reachability and only release the half-open slot.

**A timeout after the handshake is a slow host, not a dead one.** axios raises
the same `ECONNABORTED` whether the SYN went unanswered or the panel accepted
the connection and then thought for longer than the request budget (a heavy
`get_vod_info` on a busy home server). Only the former is what the guard
exists for; tripping on the latter turned "slow" into thirty seconds of
"portal is not responding" for every request, which is how users described
it. Each transport therefore reports whether the connection was established —
Electron through `ValidatedAxiosRequestConfig.onConnect` (the request gets its
own agent, observed via `observeAgentSocketConnections`, instead of the shared
keep-alive `globalAgent`; a pooled socket that is already connected counts as
connected), the web backend through `WebBackendHttpGetOptions.onConnect`,
honoured by `ProviderAxiosTransport`, which owns the `ClientRequest` — and
the accepted connection is credited THE MOMENT IT HAPPENS
(`reportGuardedHostConnected` / `reportProviderRequestConnected` from the
connect hook call `HostConnectivityGuard.reportConnected`, which clears the
failure streak but deliberately closes no open or half-open breaker: whether a
host that accepts a connection also answers is exactly what the trial exists
to find out, so the trial keeps its slot until it settles), and
`classifyHostRequestFailure(error, { connected })` then reads the eventual
host-level code as `inconclusive`. Two things hang on that
ordering. Clearing at connect time is what keeps an unanswered SYN, an
accepted-but-slow request and another unanswered SYN from adding up to a
trip, since the middle request proved the host alive in between. Crediting it
at connect time rather than when the timeout settles is what keeps a request
that connected and then hung for 30 s from reopening a breaker that later
requests opened in the meantime — by then its evidence is older than theirs.
Such a request still costs its full timeout; the guard only stops charging it
to the host. Electron does not install the observer while an environment
proxy applies to the request — decided by the very resolution axios performs
(`proxy-from-env`, the same pinned package: `<protocol>_proxy` / `all_proxy`
in either case, `no_proxy` exemptions honoured, so a LAN portal listed there
keeps its observer): through a proxy the socket connects to the proxy, whose
handshake proves nothing about the portal, so those requests keep reporting
their timeouts as host-level exactly as before. Regression coverage:
`apps/electron-backend/src/app/util/host-connectivity-guard.slow-host.spec.ts`
(real loopback sockets) and the Xtream mock's `silent` scenario, whose
`get_vod_info` / `get_series_info` accept the connection and never answer.

**A failure is only charged to the endpoint that produced it.** Reaching any
later hop _proves_ the guarded endpoint answered — the first hop is always the
URL we asked for, and only a redirect status advances the chain — so a failure
there CLEARS the guarded endpoint's record, exactly like any other response.
Merely declining to count it would leave an earlier direct failure standing, and
a single later timeout would then fast-fail an endpoint that answered in between.
Every caller passes the URL it asked for as the baseline.

**Web backend records redirect evidence explicitly.** Its `ValidatedHttpClient`
disables automatic redirects and holds one admission for the entire chain.
`ProviderRequestError.initialResponded` becomes true only after receiving an
actual redirect response. A later transport or DNS failure, URL policy refusal,
invalid location, cycle or exhausted budget therefore clears the initial
endpoint's failure streak when the chain settles. This also handles redirects
that change only the query. It does not release the half-open slot early: the
route retains ownership through validation, all hops and the final body, and
releases in `finally` even when reporting throws. Before any redirect, initial
DNS failures carry their internal resolver cause into normal classification;
policy refusals remain inconclusive. Error bodies never serialize that cause.

**Electron retains URL-based attribution.** `failedRequestUrlOf` reads
`error.request._currentUrl` first (for a following transport) and falls back to
`error.config.url` (for a native per-hop transport). The comparison is origin +
path, not the query: axios `params` can append credentials absent from the
caller's baseline. Unknown/unparseable URLs conservatively count as ordinary
failures, and query-only redirects cannot be distinguished by this fallback.
Wrapped web-backend requests use their explicit chain evidence and never infer
redirects from the transport's fixed logical hostname. The shared helper and
Electron behavior are unchanged by the web-backend fix.

Known gap: the failing hop is not guarded either (it has no token of its own),
so a permanently broken redirect chain keeps costing a full timeout.

**The key is the origin, not the host.** `URL.host` omits a default port, so
`http://panel.example` and `https://panel.example` would share one record —
two genuinely different endpoints, and a panel whose TLS listener is broken
while plain HTTP works is a routine IPTV setup. Sharing state there would let
the dead one fast-fail the working one without ever contacting it.
`URL.origin` also leaves out any `user:pass@` userinfo, so no credential
reaches the key or the log line.

Two more rules exist because of specific failure modes:

- **Siblings are not a streak.** Catalog initialization fans out three category
  requests at once; one network hiccup failing all three is one piece of
  evidence, not a trip. Each admitted request receives a guard-wide monotonic
  admission id. A counted failure saves the highest id admitted so far in that
  endpoint's failure record, not just the id of the request that failed. During
  that streak, only a request admitted beyond this boundary can add another
  failure. This distinguishes
  siblings from later attempts even if admission and completion all happen in
  the same millisecond (#1438), regardless of completion order. Admission ids
  are never reused when an endpoint's state is evicted. A recreated record has
  no failure streak, so the first old in-flight failure can still count once;
  its remaining siblings cannot add further links to that streak. Timestamps
  still determine streak expiry and cooldown. Only a failure that
  was actually counted may trip the threshold: a sibling settling after the open
  window elapsed would otherwise start a fresh one off the existing count and
  push the half-open trial past the intended cooldown.
- **A reset invalidates reports already in flight.** `reset()` bumps a per-host
  epoch instead of deleting the record, and a failure reported under an older
  epoch is discarded. Without that, the 30-second stragglers a user was waiting
  behind settle right after they press Retry and re-open the breaker underneath
  the very retry that cleared it.

### Trial ownership follows the request lifetime

A half-open slot has no wall-clock expiry (#1439). A redirect chain gives each
hop a separate timeout, and a body that keeps delivering bytes can exceed any
fixed deadline without reaching its inactivity timeout. Neither may admit a
second trial while the first request remains pending.

The four request owners (Electron Xtream/Stalker IPC and web-backend
`/xtream`/`/stalker`) acquire inside `try`, await the whole transport operation,
report its outcome, and unconditionally release any remaining token in `finally`.
`reportInconclusive` is the idempotent cleanup operation: it releases only the
matching `trialId` and epoch and records no reachability evidence. This is the
leak backstop even when debug logging, classification, or response formatting
throws before an outcome report. Cleanup also runs while the environment kill
switch is on, so switching it back off cannot revive a completed trial's slot.
Desktop preference transitions invalidate the old guard and tokens as before.

Cancellation releases after the awaited transport rejects, when it actually
settles, rather than merely when an abort is requested. Xtream's AbortSignal
continues through all Electron redirect hops. Stalker has no IPC cancellation
signal, and closing a PWA client connection does not cancel the backend's outbound
request; those slots remain owned until that transport settles. The transport's
existing timeout handles inactivity. No heartbeat, polling timer, or larger
trial deadline is needed. A future caller must preserve the `try`/`finally`
contract; a custom transport that never settles needs cancellation at its own
layer and must not be silently overlapped by a second trial.

The slot still has an identity: delayed cleanup from a released owner must not
free its replacement. A late failure still follows the ordinary streak and epoch
rules; a real HTTP response still proves reachability and clears the record.
Explicit reset, discovery success, preference transitions and bounded state
eviction retain their existing rules for forgetting evidence.

## The fast-fail error is a renderer contract

`HostConnectivityGuardError` is a real `Error`: Electron serializes a rejected
plain object to `[object Object]`, which would destroy the renderer's
classification. It carries no `status` property, because
`getStalkerRequestErrorStatus` reads that field first.

The message comes from `buildHostConnectivityFastFailMessage()` in
`libs/shared/interfaces/src/lib/host-connectivity.util.ts`. It names the full
endpoint, scheme included, so a user who imported the same panel over both HTTP
and HTTPS can tell which one was skipped. Its wording is load-bearing — the
Stalker renderer classifies transport failures purely from message text:

| Must not contain                                                                                     | Otherwise                                                                                                                           |
| ---------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------- |
| `HTTP Error <code>`                                                                                  | reads as "endpoint absent, probe the next candidate"; 404/401/403 also fire lazy portal repair against a host we just declared dead |
| `timed out`, `timeout of Nms`, `ETIMEDOUT`                                                           | discovery walks every candidate instead of aborting early                                                                           |
| `authorization`, `unauthorized`, `access denied`, `invalid token`, `auth failed`, `handshake failed` | fires lazy portal repair (a bare `authorization` matches)                                                                           |

What remains is the "connection-level failure" slot the renderer already has for
ECONNREFUSED/ENOTFOUND: discovery stops probing and reports the host
unreachable, `shouldAttemptRepair` returns false, and the message reaches error
snackbars verbatim — which is why it reads like a sentence and names the
endpoint.

**The endpoint in that sentence is user data, so the marker outranks the
heuristics.** A portal at `https://authorization.example` would otherwise make
its own fast-fail message match the broad auth phrase set and send an
unreachable host into lazy portal repair. `isStalkerAuthFailureMessage` and
`isStalkerProbeTimeout` therefore both return false for a message
`isHostConnectivityFastFailMessage` recognises, before their phrase matching
runs. (`getStalkerRequestErrorStatus` needs no such guard: `HTTP Error <code>`
contains a space, which a hostname cannot.)
`stalker-portal-discovery.utils.spec.ts` pins all three properties for the bare
and the IPC-wrapped form.

## Exemption: endpoint discovery

`STALKER_REQUEST` accepts `skipConnectionGuard`, set **only** by
`StalkerPortalDiscoveryService.probeContent`. Semantics: bypass the check, never
count failures, **but still report successes.**

Discovery walks several candidate paths on one host and expects most of them to
fail; counting that would let it declare a slow-but-alive portal unreachable.
Reporting successes is equally load-bearing: `confirmFullPortal` runs the full
non-exempt authentication flow against auth-gated candidates, so without it two
hung handshakes could open the breaker mid-discovery and the next authenticate
would fast-fail into the "host unreachable" slot — abandoning a portal that
works. "Success" here means any observed response, including one attached to a
rejection: `validateStatus` lets 4xx through but rejects 5xx with
`error.response` set, and a 5xx proves the origin answered just as well as a
body does. That is why the exempt path reports through
`reportGuardedHostFailure(token, error, { countFailures: false })` rather than
skipping the report.

The flag has to survive the PWA transport too, or discovery is exempt on the
desktop and policed on the web. `PwaService.forwardStalkerRequest` forwards it
as a `/stalker` control param and the route applies the same semantics — and,
like `macAddress`/`token`/`serialNumber`, it is stripped from the query
forwarded to the portal, because it is our control flag and not protocol
content.

## Explicit reset

`CONNECTIVITY_GUARD_RESET` (`{ url }`) is handled by
`apps/electron-backend/src/app/events/connectivity-guard.events.ts`. One key
derivation is enough: both `normalizeXtreamServerUrl` and
`buildStalkerRequestUrl` rebuild their request URL from `URL.origin`, so the
origin a request ends up using is always the origin of the URL stored on the
playlist.

**The rule: every user-driven retry or refresh that issues portal requests must
reset the guard before its first request.** The failures that opened the breaker
are usually the very ones the user is retrying, so a reset placed after the
request — or missing — makes the affordance do nothing until the window expires.
Automatic and first-load paths deliberately do NOT reset: only a user action
means "contact this host now", and clearing evidence the guard just collected
would defeat it.

Call sites:

- `retryContentInitialization` (`with-content.feature.ts`) — the Xtream
  content-gate Retry button. The reset is the **first** awaited statement, before
  the portal status check: a tripped guard fast-fails that check, its
  `unavailable` verdict returns early, and a reset placed any later would never
  run.
- `StalkerPortalDiscoveryService.discover()` — one site covering import, the Edit
  dialog and lazy repair, which also guarantees a freshly edited address never
  inherits a refusal recorded for the previous one.
- `retryContentPage` (`with-stalker-content.feature.ts`) — the Stalker grid
  tail's append retry.
- `StalkerSearchComponent`'s search-page retry — the search results have their
  own append error and retry, separate from the catalog's.
- `StalkerItvCacheService.refresh()` — the Live TV refresh button, on the same
  path that already clears the cache's own error cooldown. This also covers
  `refreshChannels()` in the live layout.
- Both account-info dialogs' Retry buttons (`AccountInfoComponent.reload()` for
  Xtream, `StalkerAccountInfoComponent.reload()` for Stalker). Their automatic
  load on open goes through a private `load()` that does not reset.
- The destructive Xtream refresh — before anything is deleted. It removes the
  cached catalog and then bootstraps a re-import whose status request an open
  guard would fast-fail, leaving the user with no catalog at all until the
  cooldown expires. The reset lives in `XtreamRefreshFlowService.runRefresh()`
  (`libs/playlist/shared/ui`), which owns the whole flow for both of its entry
  points: `PlaylistRefreshActionService.refreshXtream()` and
  `RecentPlaylistsComponent.refreshXtreamPlaylist()` (the Workspace sources
  page). Those two used to be independent near-duplicates, which is exactly why
  the second one was missed first time round; they now differ only in the
  `XtreamRefreshProgressReporter` they hand over, and a reporter cannot reach
  the reset. Keep it that way — a third entry point should pass a reporter, not
  copy the sequence.
- `PortalStatusService.checkPortalStatusDetails` when `skipCache` is set — the
  user-initiated "Test Connection".

Two retry paths deliberately have no reset: Xtream's `retryAppend()` is a no-op
because in-memory appends cannot fail, and the guard's own half-open trial is not
a user action.

Where a retry clears a UI error flag, that flag is cleared **synchronously**
before awaiting the reset — otherwise the retry branch stays re-enterable and
the next `nearEnd` event fires a second retry.

They all go through `resetHostConnectivityGuard()`
(`libs/services/src/lib/host-connectivity-reset.ts`), which holds the one rule
they share: the reset is best effort, because the guard only ever _delays_ a
request and a failed reset must not block the action that asked for it. Both
runtimes honour it, so neither keeps fast-failing an endpoint the user just
asked to retry — in the PWA `PwaService` forwards it to the web backend's
`POST /connectivity-guard/reset`, since the breaker lives in the backend
process and a local reset would clear nothing. That route takes the raw
provider URL rather than a registered `targetId`, because callers reset
precisely when the address may have changed, which is before any target exists
for it; it fetches nothing, reads only the origin, and never logs the URL.

## Interaction with VOD multi-source

No exemption is needed. Multi-source resolution reaches `get_vod_info` over
`XTREAM_REQUEST`, but `VodSourceResolverService.loadVodDetails` already catches
that failure and falls back to its learned container cache or `null`, and
`probeSource` maps `null` to the verdict `unknown` — which
`VodSourceProbeCacheService` deliberately does not cache. A guard fast-fail
therefore degrades to a retryable "could not check", exactly matching the
module's contract that unreachable ≠ contacted-and-refused. The
`STREAM_PROBE_URL` reachability half never rejects and is untouched.

## Tests

- `libs/shared/host-health/src/lib/host-connectivity-guard.spec.ts` — the state
  machine, with an injected clock, long-lived trial ownership, late cleanup,
  and settlement while the environment kill switch is on.
- `apps/electron-backend/src/app/util/host-connectivity-guard.spec.ts` — the
  main-process singleton and redirect attribution through it.
- `apps/web-backend/src/app/web-backend-app.host-guard.spec.ts` — the proxy
  routes: the HTTP 200 refusal shape, that the outbound request really is
  skipped, that an open endpoint is refused before a DNS lookup is spent on it,
  redirect attribution, the discovery exemption, half-open, the reset endpoint,
  and that playlist/EPG downloads are never fast-failed. The per-route timeouts
  are asserted in `web-backend-app.spec.ts`, which pins the whole outbound
  request shape.
- `apps/web/src/app/services/pwa.service.spec.ts` — that a fast-fail reaches the
  renderer with no numeric `status` and no `HTTP Error <code>` in its message,
  that the discovery bypass is forwarded, and that a reset calls the backend.
- `apps/electron-backend/src/app/events/stalker.events.spec.ts` and
  `xtream.events.spec.ts` — trip, fast-fail without contacting axios, reset,
  exemption, long pending requests and redirect chains, cancellation, cleanup
  when debug reporting throws, and the absence of per-request log spam.
- `libs/portal/stalker/data-access/src/lib/stalker-portal-discovery.utils.spec.ts`
  and `stalker-portal-repair.service.spec.ts` — the message contract.