iptvnator/docs/architecture/host-connectivity-guard.md

# Host Connectivity Guard

Per-host circuit breaker for portal requests, in both processes that make them:
the Electron main process and the self-hosted web backend.

## The problem

Every request to an unreachable portal costs its full axios timeout — 30 s for
`XTREAM_REQUEST`, 15 s for `STALKER_REQUEST` (30 s for `create_link`), and the
same budgets on the web backend's `/xtream` and `/stalker` routes. Browsing a
dead portal's catalog issues dozens of those back to back, which shows up as
30-second spinners and a log full of identical failures. Once a host has refused
to answer twice in a row there is nothing left to learn from waiting again.

The web backend had a worse version of the same problem first: its proxy routes
passed no `timeout` at all, so a provider that accepted a connection and then
went silent held the request until the OS gave up on the TCP connection. A
breaker is only useful once _not answering_ is bounded, which is why the timeouts
and the breaker landed there together.

## Where it lives

`libs/shared/host-health` (`@iptvnator/shared/host-health`, tagged
`scope:shared` / `domain:shared-runtime` / `type:util`) — the breaker class, the
failure classification and the redirect-attribution helpers, with no transport,
logger or process singleton of its own. The owning app supplies the clock and
decides how many guards exist.

Each runtime owns its instance:

- **Electron** — `apps/electron-backend/src/app/util/host-connectivity-guard.ts`
  holds one guard for the whole process, wired into both IPC handlers. The
  handlers are the choke point that sees _all_ traffic to a host. The renderer's
  `executeStalkerRequest` is not: it has four documented bypasses (auth,
  endpoint discovery, account info, the row-less stream resolver), and that is
  exactly the traffic that hits dead hosts.
- **PWA** — `apps/web-backend/src/app/host-guard.ts` guards the `/xtream` and
  `/stalker` proxy routes. The guard instance is injected through
  `WebBackendAppOptions.hostGuard`, alongside `now` and `guid`, so specs drive
  it with a clock they own.

### Request timeouts are the precondition

A breaker keyed on "the endpoint did not answer" is only useful once _not
answering_ is bounded. `apps/web-backend` originally passed no `timeout` at all,
so a silent host hung on OS-level TCP timeouts. Both runtimes now use the same
budgets: Xtream 30 s, Stalker 15 s (30 s for `create_link`, which mints a stream
URL before answering), playlist and XMLTV downloads 30 s.

Those numbers are safe for large downloads. On axios' default
(follow-redirects) transport `timeout` is **not** a wall-clock deadline for the
whole response: it bounds the time to response headers and then continues as the
socket's inactivity timeout for the body. A multi-megabyte XMLTV file that keeps
delivering bytes is never cut off mid-transfer — only a stalled one is.

### Scope: portal calls only

Both runtimes guard the portal API paths and nothing else. Playlist and XMLTV
downloads (`/parse`, `/parse-xml`, and their Electron equivalents) get the
timeouts but not the breaker, deliberately:

- The problem being solved is a catalog fan-out — dozens of requests to one
  endpoint back to back. A download is a single request.
- A download is usually the direct result of the user asking for it (the
  add-playlist dialog sends `PLAYLIST_PARSE_BY_URL`). Refusing an immediate
  retry is a regression, not a protection, and there is no natural reset site on
  that path the way portal Retry has one.
- A large XMLTV transfer can legitimately run for minutes — the timeout is
  idle-based, see above — which outlives the 45 s half-open trial expiry and
  would let a second trial in behind the first.

## Rules

Being wrong here means refusing to talk to a portal that works, so every rule
errs towards contacting the host:

|             |                                                                        |
| ----------- | ---------------------------------------------------------------------- |
| Trip        | 2 consecutive host-level failures within an inclusive 120 s window     |
| Open for    | 30 s (`OPEN_DURATION_MS`), matching the repo's other cooldowns         |
| Half-open   | exactly ONE trial request; the rest keep fast-failing until it settles |
| Reset       | any HTTP response — 200, 404, even 502 — the host answered             |
| Key         | `URL.origin` — scheme, host **and** port (see below)                   |
| Kill switch | `IPTVNATOR_DISABLE_CONNECTIVITY_GUARD=1` (read per call)               |

**Host-level failure** means an error with no HTTP response whose code is one of
`ETIMEDOUT`, `ECONNABORTED`, `ENOTFOUND`, `EAI_AGAIN`, `ECONNREFUSED`,
`EHOSTUNREACH`, `ENETUNREACH`. `ECONNRESET` is deliberately excluded: a reset
mid-transfer happens on hosts that are very much alive. Cancelled requests
(`ERR_CANCELED`) and SSRF-policy refusals are `inconclusive` — they say nothing
about reachability and only release the half-open slot.

**A failure is only charged to the endpoint that produced it.** Reaching any
later hop _proves_ the guarded endpoint answered — the first hop is always the
URL we asked for, and only a redirect status advances the chain — so a failure
there CLEARS the guarded endpoint's record, exactly like any other response.
Merely declining to count it would leave an earlier direct failure standing, and
a single later timeout would then fast-fail an endpoint that answered in between.
Every caller passes the URL it asked for as the baseline.

**Where the failed hop is found depends on the transport, and both are in play.**

| | Electron | Web backend |
| --- | --- | --- |
| Redirects | followed hop by hop (`maxRedirects: 0`), each its own request | followed inside one request by follow-redirects |
| Failed hop is in | `error.config.url` | `error.request._currentUrl` |

`failedRequestUrlOf` reads `_currentUrl` first and falls back to `config.url`,
which is correct for both: a per-hop request exposes no `_currentUrl`, and on the
following transport `config` is built once and keeps the URL we asked for — so
reading `config.url` there would compare a URL with itself, find no redirect, and
charge a dead destination to the provider that answered. Anything added here must
work on both, because the same helper serves both.

**The comparison is origin + path, not the whole URL.** A same-origin redirect
(`/player_api.php` → `/slow/player_api.php`) proves the endpoint answered just as
much as a cross-origin one, and charging it would fast-fail every OTHER call to a
portal that answers — so the path has to be part of it. The query must NOT be:
the web backend passes Xtream credentials through axios' `params`, so the sent
URL always carries a query the baseline does not, and comparing whole URLs made
every ordinary failure look like a redirect and stopped the breaker from ever
opening. What that gives up is a redirect that changes nothing but the query,
which is then counted as an ordinary failure — the safe direction.

It requires positive evidence — anything unparseable or unknown counts the
failure as usual, because guessing "redirect" here would stop the guard from ever
tripping — and a failure that names no URL at all is still counted.

Known gap: the failing hop is not guarded either (it has no token of its own),
so a permanently broken redirect chain keeps costing a full timeout.

**The key is the origin, not the host.** `URL.host` omits a default port, so
`http://panel.example` and `https://panel.example` would share one record —
two genuinely different endpoints, and a panel whose TLS listener is broken
while plain HTTP works is a routine IPTV setup. Sharing state there would let
the dead one fast-fail the working one without ever contacting it.
`URL.origin` also leaves out any `user:pass@` userinfo, so no credential
reaches the key or the log line.

Two more rules exist because of specific failure modes:

- **Siblings are not a streak.** Catalog initialization fans out three category
  requests at once; one network hiccup failing all three is one piece of
  evidence, not a trip. A failure counts only if its request started at or after
  the moment the previous failure was recorded. Timestamps are millisecond
  coarse, so this only separates siblings once a request actually took time —
  which is precisely the expensive case worth protecting. Only a failure that
  was actually counted may trip the threshold: a sibling settling after the open
  window elapsed would otherwise start a fresh one off the existing count and
  push the half-open trial past the intended cooldown.
- **A reset invalidates reports already in flight.** `reset()` bumps a per-host
  epoch instead of deleting the record, and a failure reported under an older
  epoch is discarded. Without that, the 30-second stragglers a user was waiting
  behind settle right after they press Retry and re-open the breaker underneath
  the very retry that cleared it.

A half-open trial that never reports back expires after 45 s, so a leaked token
cannot leave the breaker open forever. That expiry is why the slot has an
identity: a trial can genuinely outlive its window — `requestWithValidatedRedirects`
gives each of up to five redirect hops its own 30 s budget — and once a
replacement has been admitted, the abandoned request's late report must not free
the replacement's slot and let a third request through. `trial: true` alone
cannot tell the two apart, so the token carries the slot id it owns; an
abandoned trial's failure is still counted as ordinary evidence.

## The fast-fail error is a renderer contract

`HostConnectivityGuardError` is a real `Error`: Electron serializes a rejected
plain object to `[object Object]`, which would destroy the renderer's
classification. It carries no `status` property, because
`getStalkerRequestErrorStatus` reads that field first.

The message comes from `buildHostConnectivityFastFailMessage()` in
`libs/shared/interfaces/src/lib/host-connectivity.util.ts`. It names the full
endpoint, scheme included, so a user who imported the same panel over both HTTP
and HTTPS can tell which one was skipped. Its wording is load-bearing — the
Stalker renderer classifies transport failures purely from message text:

| Must not contain                                                                                     | Otherwise                                                                                                                           |
| ---------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------- |
| `HTTP Error <code>`                                                                                  | reads as "endpoint absent, probe the next candidate"; 404/401/403 also fire lazy portal repair against a host we just declared dead |
| `timed out`, `timeout of Nms`, `ETIMEDOUT`                                                           | discovery walks every candidate instead of aborting early                                                                           |
| `authorization`, `unauthorized`, `access denied`, `invalid token`, `auth failed`, `handshake failed` | fires lazy portal repair (a bare `authorization` matches)                                                                           |

What remains is the "connection-level failure" slot the renderer already has for
ECONNREFUSED/ENOTFOUND: discovery stops probing and reports the host
unreachable, `shouldAttemptRepair` returns false, and the message reaches error
snackbars verbatim — which is why it reads like a sentence and names the
endpoint.

**The endpoint in that sentence is user data, so the marker outranks the
heuristics.** A portal at `https://authorization.example` would otherwise make
its own fast-fail message match the broad auth phrase set and send an
unreachable host into lazy portal repair. `isStalkerAuthFailureMessage` and
`isStalkerProbeTimeout` therefore both return false for a message
`isHostConnectivityFastFailMessage` recognises, before their phrase matching
runs. (`getStalkerRequestErrorStatus` needs no such guard: `HTTP Error <code>`
contains a space, which a hostname cannot.)
`stalker-portal-discovery.utils.spec.ts` pins all three properties for the bare
and the IPC-wrapped form.

## Exemption: endpoint discovery

`STALKER_REQUEST` accepts `skipConnectionGuard`, set **only** by
`StalkerPortalDiscoveryService.probeContent`. Semantics: bypass the check, never
count failures, **but still report successes.**

Discovery walks several candidate paths on one host and expects most of them to
fail; counting that would let it declare a slow-but-alive portal unreachable.
Reporting successes is equally load-bearing: `confirmFullPortal` runs the full
non-exempt authentication flow against auth-gated candidates, so without it two
hung handshakes could open the breaker mid-discovery and the next authenticate
would fast-fail into the "host unreachable" slot — abandoning a portal that
works. "Success" here means any observed response, including one attached to a
rejection: `validateStatus` lets 4xx through but rejects 5xx with
`error.response` set, and a 5xx proves the origin answered just as well as a
body does. That is why the exempt path reports through
`reportGuardedHostFailure(token, error, { countFailures: false })` rather than
skipping the report.

The flag has to survive the PWA transport too, or discovery is exempt on the
desktop and policed on the web. `PwaService.forwardStalkerRequest` forwards it
as a `/stalker` control param and the route applies the same semantics — and,
like `macAddress`/`token`/`serialNumber`, it is stripped from the query
forwarded to the portal, because it is our control flag and not protocol
content.

## Explicit reset

`CONNECTIVITY_GUARD_RESET` (`{ url }`) is handled by
`apps/electron-backend/src/app/events/connectivity-guard.events.ts`. One key
derivation is enough: both `normalizeXtreamServerUrl` and
`buildStalkerRequestUrl` rebuild their request URL from `URL.origin`, so the
origin a request ends up using is always the origin of the URL stored on the
playlist.

**The rule: every user-driven retry or refresh that issues portal requests must
reset the guard before its first request.** The failures that opened the breaker
are usually the very ones the user is retrying, so a reset placed after the
request — or missing — makes the affordance do nothing until the window expires.
Automatic and first-load paths deliberately do NOT reset: only a user action
means "contact this host now", and clearing evidence the guard just collected
would defeat it.

Call sites:

- `retryContentInitialization` (`with-content.feature.ts`) — the Xtream
  content-gate Retry button. The reset is the **first** awaited statement, before
  the portal status check: a tripped guard fast-fails that check, its
  `unavailable` verdict returns early, and a reset placed any later would never
  run.
- `StalkerPortalDiscoveryService.discover()` — one site covering import, the Edit
  dialog and lazy repair, which also guarantees a freshly edited address never
  inherits a refusal recorded for the previous one.
- `retryContentPage` (`with-stalker-content.feature.ts`) — the Stalker grid
  tail's append retry.
- `StalkerSearchComponent`'s search-page retry — the search results have their
  own append error and retry, separate from the catalog's.
- `StalkerItvCacheService.refresh()` — the Live TV refresh button, on the same
  path that already clears the cache's own error cooldown. This also covers
  `refreshChannels()` in the live layout.
- Both account-info dialogs' Retry buttons (`AccountInfoComponent.reload()` for
  Xtream, `StalkerAccountInfoComponent.reload()` for Stalker). Their automatic
  load on open goes through a private `load()` that does not reset.
- The destructive Xtream refresh — before anything is deleted. It removes the
  cached catalog and then bootstraps a re-import whose status request an open
  guard would fast-fail, leaving the user with no catalog at all until the
  cooldown expires. The reset lives in `XtreamRefreshFlowService.runRefresh()`
  (`libs/playlist/shared/ui`), which owns the whole flow for both of its entry
  points: `PlaylistRefreshActionService.refreshXtream()` and
  `RecentPlaylistsComponent.refreshXtreamPlaylist()` (the Workspace sources
  page). Those two used to be independent near-duplicates, which is exactly why
  the second one was missed first time round; they now differ only in the
  `XtreamRefreshProgressReporter` they hand over, and a reporter cannot reach
  the reset. Keep it that way — a third entry point should pass a reporter, not
  copy the sequence.
- `PortalStatusService.checkPortalStatusDetails` when `skipCache` is set — the
  user-initiated "Test Connection".

Two retry paths deliberately have no reset: Xtream's `retryAppend()` is a no-op
because in-memory appends cannot fail, and the guard's own half-open trial is not
a user action.

Where a retry clears a UI error flag, that flag is cleared **synchronously**
before awaiting the reset — otherwise the retry branch stays re-enterable and
the next `nearEnd` event fires a second retry.

They all go through `resetHostConnectivityGuard()`
(`libs/services/src/lib/host-connectivity-reset.ts`), which holds the one rule
they share: the reset is best effort, because the guard only ever _delays_ a
request and a failed reset must not block the action that asked for it. Both
runtimes honour it, so neither keeps fast-failing an endpoint the user just
asked to retry — in the PWA `PwaService` forwards it to the web backend's
`POST /connectivity-guard/reset`, since the breaker lives in the backend
process and a local reset would clear nothing. That route takes the raw
provider URL rather than a registered `targetId`, because callers reset
precisely when the address may have changed, which is before any target exists
for it; it fetches nothing, reads only the origin, and never logs the URL.

## Interaction with VOD multi-source

No exemption is needed. Multi-source resolution reaches `get_vod_info` over
`XTREAM_REQUEST`, but `VodSourceResolverService.loadVodDetails` already catches
that failure and falls back to its learned container cache or `null`, and
`probeSource` maps `null` to the verdict `unknown` — which
`VodSourceProbeCacheService` deliberately does not cache. A guard fast-fail
therefore degrades to a retryable "could not check", exactly matching the
module's contract that unreachable ≠ contacted-and-refused. The
`STREAM_PROBE_URL` reachability half never rejects and is untouched.

## Tests

- `libs/shared/host-health/src/lib/host-connectivity-guard.spec.ts` — the state
  machine, with an injected clock.
- `apps/electron-backend/src/app/util/host-connectivity-guard.spec.ts` — the
  main-process singleton and redirect attribution through it.
- `apps/web-backend/src/app/web-backend-app.host-guard.spec.ts` — the proxy
  routes: the HTTP 200 refusal shape, that the outbound request really is
  skipped, that an open endpoint is refused before a DNS lookup is spent on it,
  redirect attribution, the discovery exemption, half-open, the reset endpoint,
  and that playlist/EPG downloads are never fast-failed. The per-route timeouts
  are asserted in `web-backend-app.spec.ts`, which pins the whole outbound
  request shape.
- `apps/web/src/app/services/pwa.service.spec.ts` — that a fast-fail reaches the
  renderer with no numeric `status` and no `HTTP Error <code>` in its message,
  that the discovery bypass is forwarded, and that a reset calls the backend.
- `apps/electron-backend/src/app/events/stalker.events.spec.ts` and
  `xtream.events.spec.ts` — trip, fast-fail without contacting axios, reset,
  exemption, and the absence of per-request log spam.
- `libs/portal/stalker/data-access/src/lib/stalker-portal-discovery.utils.spec.ts`
  and `stalker-portal-repair.service.spec.ts` — the message contract.