# Host Connectivity Guard Per-host circuit breaker for portal requests, in both processes that make them: the Electron main process and the self-hosted web backend. ## The problem Every request to an unreachable portal costs its full axios timeout — 30 s for `XTREAM_REQUEST`, 15 s for `STALKER_REQUEST` (30 s for `create_link`), and the same budgets on the web backend's `/xtream` and `/stalker` routes. Browsing a dead portal's catalog issues dozens of those back to back, which shows up as 30-second spinners and a log full of identical failures. Once a host has refused to answer twice in a row there is nothing left to learn from waiting again. The web backend had a worse version of the same problem first: its proxy routes passed no `timeout` at all, so a provider that accepted a connection and then went silent held the request until the OS gave up on the TCP connection. A breaker is only useful once _not answering_ is bounded, which is why the timeouts and the breaker landed there together. ## Where it lives `libs/shared/host-health` (`@iptvnator/shared/host-health`, tagged `scope:shared` / `domain:shared-runtime` / `type:util`) — the breaker class, the failure classification and the redirect-attribution helpers, with no transport, logger or process singleton of its own. The owning app supplies the clock and decides how many guards exist. Each runtime owns its instance: - **Electron** — `apps/electron-backend/src/app/util/host-connectivity-guard.ts` holds one guard for the whole process, wired into both IPC handlers. The handlers are the choke point that sees _all_ traffic to a host. The renderer's `executeStalkerRequest` is not: it has four documented bypasses (auth, endpoint discovery, account info, the row-less stream resolver), and that is exactly the traffic that hits dead hosts. - **PWA** — `apps/web-backend/src/app/host-guard.ts` guards the `/xtream` and `/stalker` proxy routes. The guard instance is injected through `WebBackendAppOptions.hostGuard`, alongside `now` and `guid`, so specs drive it with a clock they own. ### Request timeouts are the precondition A breaker keyed on "the endpoint did not answer" is only useful once _not answering_ is bounded. `apps/web-backend` originally passed no `timeout` at all, so a silent host hung on OS-level TCP timeouts. Both runtimes now use the same budgets: Xtream 30 s, Stalker 15 s (30 s for `create_link`, which mints a stream URL before answering), playlist and XMLTV downloads 30 s. Those numbers are safe for large downloads. On axios 1.20's native HTTP transport (`maxRedirects: 0`) each hop's `timeout` is **not** a wall-clock deadline for the whole response: it bounds the time to response headers and then continues as the socket's inactivity timeout for the body. A multi-megabyte XMLTV file that keeps delivering bytes is never cut off mid-transfer — only a stalled one is. The web backend explicitly restores socket inactivity handling while reading the final stream response, because axios stream mode stops its timeout handler at headers. Truncated, malformed and timed-out final bodies retain their received response status for reachability classification; cancellation retains its own semantics. ### Scope: portal calls only Both runtimes guard the portal API paths and nothing else. Playlist and XMLTV downloads (`/parse`, `/parse-xml`, and their Electron equivalents) get the timeouts but not the breaker, deliberately: - The problem being solved is a catalog fan-out — dozens of requests to one endpoint back to back. A download is a single request. - A download is usually the direct result of the user asking for it (the add-playlist dialog sends `PLAYLIST_PARSE_BY_URL`). Refusing an immediate retry is a regression, not a protection, and there is no natural reset site on that path the way portal Retry has one. ## Desktop preference and account feedback Settings > General > Portal connections offers **Pause requests to unavailable portals**, enabled by default. `Settings.portalConnectivityGuard` is default-on for missing or non-boolean legacy values; only explicit `false` opts out. The form stages edits until Save and persists the choice through the normal settings store. `SETTINGS_UPDATE` mirrors it to Electron's `PORTAL_CONNECTIVITY_GUARD` config key and applies it immediately. `SettingsEvents.bootstrapSettingsEvents` loads that mirror before the renderer is loaded, so a saved opt-out already applies to the first portal request after restarting. The preference controls both Xtream and Stalker. A real preference transition replaces the Electron guard and its weak set of admitted request tokens, clearing existing cooldowns and ignoring completions from the old generation. Disabled requests return no token and cannot contribute failures when the user re-enables the guard. Saving an unchanged value preserves existing evidence. `IPTVNATOR_DISABLE_CONNECTIVITY_GUARD=1` (or `true`) still overrides an enabled preference. The UI is capability-gated by `supportsPortalConnectivityGuard` (`updateSettings` plus `resetHostConnectivityGuard`); the PWA has no client-side switch for its shared server-wide guard. Both account-info dialogs classify the existing cross-process guard message and show a localized **Requests temporarily paused** explanation with **Retry now**. Stalker keeps cached account data visible and offers the same retry beside the paused refresh notice. Retry resets before requesting again; actual network failures retain the generic unavailable state. These notices describe the last request outcome, not a live countdown. Same-millisecond sibling counting is handled by monotonic admission ids (#1438). ## Rules Being wrong here means refusing to talk to a portal that works, so every rule errs towards contacting the host: | | | | ----------- | ---------------------------------------------------------------------- | | Trip | 2 consecutive host-level failures within an inclusive 120 s window | | Open for | 30 s (`OPEN_DURATION_MS`), matching the repo's other cooldowns | | Half-open | exactly ONE trial request; the rest keep fast-failing until it settles | | Reset | any HTTP response — 200, 404, even 502; an accepted TCP connection clears the streak but not an open breaker | | Key | `URL.origin` — scheme, host **and** port (see below) | | Kill switch | `IPTVNATOR_DISABLE_CONNECTIVITY_GUARD=1` (read per call) | **Host-level failure** means an error with no HTTP response whose code is one of `ETIMEDOUT`, `ECONNABORTED`, `ENOTFOUND`, `EAI_AGAIN`, `ECONNREFUSED`, `EHOSTUNREACH`, `ENETUNREACH` — observed **before the TCP connection was established**. `ECONNRESET` is deliberately excluded: a reset mid-transfer happens on hosts that are very much alive. Cancelled requests (`ERR_CANCELED`) and SSRF-policy refusals are `inconclusive` — they say nothing about reachability and only release the half-open slot. **A timeout after the handshake is a slow host, not a dead one.** axios raises the same `ECONNABORTED` whether the SYN went unanswered or the panel accepted the connection and then thought for longer than the request budget (a heavy `get_vod_info` on a busy home server). Only the former is what the guard exists for; tripping on the latter turned "slow" into thirty seconds of "portal is not responding" for every request, which is how users described it. Each transport therefore reports whether the connection was established — Electron through `ValidatedAxiosRequestConfig.onConnect` (the request gets its own agent, observed via `observeAgentSocketConnections`, instead of the shared keep-alive `globalAgent`; a pooled socket that is already connected counts as connected), the web backend through `WebBackendHttpGetOptions.onConnect`, honoured by `ProviderAxiosTransport`, which owns the `ClientRequest` — and the accepted connection is credited THE MOMENT IT HAPPENS (`reportGuardedHostConnected` / `reportProviderRequestConnected` from the connect hook call `HostConnectivityGuard.reportConnected`, which clears the failure streak but deliberately closes no open or half-open breaker: whether a host that accepts a connection also answers is exactly what the trial exists to find out, so the trial keeps its slot until it settles), and `classifyHostRequestFailure(error, { connected })` then reads the eventual host-level code as `inconclusive`. Two things hang on that ordering. Clearing at connect time is what keeps an unanswered SYN, an accepted-but-slow request and another unanswered SYN from adding up to a trip, since the middle request proved the host alive in between. Crediting it at connect time rather than when the timeout settles is what keeps a request that connected and then hung for 30 s from reopening a breaker that later requests opened in the meantime — by then its evidence is older than theirs. Such a request still costs its full timeout; the guard only stops charging it to the host. Electron does not install the observer while an environment proxy applies to the request — decided by the very resolution axios performs (`proxy-from-env`, the same pinned package: `_proxy` / `all_proxy` in either case, `no_proxy` exemptions honoured, so a LAN portal listed there keeps its observer): through a proxy the socket connects to the proxy, whose handshake proves nothing about the portal, so those requests keep reporting their timeouts as host-level exactly as before. Regression coverage: `apps/electron-backend/src/app/util/host-connectivity-guard.slow-host.spec.ts` (real loopback sockets) and the Xtream mock's `silent` scenario, whose `get_vod_info` / `get_series_info` accept the connection and never answer. **A failure is only charged to the endpoint that produced it.** Reaching any later hop _proves_ the guarded endpoint answered — the first hop is always the URL we asked for, and only a redirect status advances the chain — so a failure there CLEARS the guarded endpoint's record, exactly like any other response. Merely declining to count it would leave an earlier direct failure standing, and a single later timeout would then fast-fail an endpoint that answered in between. Every caller passes the URL it asked for as the baseline. **Web backend records redirect evidence explicitly.** Its `ValidatedHttpClient` disables automatic redirects and holds one admission for the entire chain. `ProviderRequestError.initialResponded` becomes true only after receiving an actual redirect response. A later transport or DNS failure, URL policy refusal, invalid location, cycle or exhausted budget therefore clears the initial endpoint's failure streak when the chain settles. This also handles redirects that change only the query. It does not release the half-open slot early: the route retains ownership through validation, all hops and the final body, and releases in `finally` even when reporting throws. Before any redirect, initial DNS failures carry their internal resolver cause into normal classification; policy refusals remain inconclusive. Error bodies never serialize that cause. **Electron retains URL-based attribution.** `failedRequestUrlOf` reads `error.request._currentUrl` first (for a following transport) and falls back to `error.config.url` (for a native per-hop transport). The comparison is origin + path, not the query: axios `params` can append credentials absent from the caller's baseline. Unknown/unparseable URLs conservatively count as ordinary failures, and query-only redirects cannot be distinguished by this fallback. Wrapped web-backend requests use their explicit chain evidence and never infer redirects from the transport's fixed logical hostname. The shared helper and Electron behavior are unchanged by the web-backend fix. Known gap: the failing hop is not guarded either (it has no token of its own), so a permanently broken redirect chain keeps costing a full timeout. **The key is the origin, not the host.** `URL.host` omits a default port, so `http://panel.example` and `https://panel.example` would share one record — two genuinely different endpoints, and a panel whose TLS listener is broken while plain HTTP works is a routine IPTV setup. Sharing state there would let the dead one fast-fail the working one without ever contacting it. `URL.origin` also leaves out any `user:pass@` userinfo, so no credential reaches the key or the log line. Two more rules exist because of specific failure modes: - **Siblings are not a streak.** Catalog initialization fans out three category requests at once; one network hiccup failing all three is one piece of evidence, not a trip. Each admitted request receives a guard-wide monotonic admission id. A counted failure saves the highest id admitted so far in that endpoint's failure record, not just the id of the request that failed. During that streak, only a request admitted beyond this boundary can add another failure. This distinguishes siblings from later attempts even if admission and completion all happen in the same millisecond (#1438), regardless of completion order. Admission ids are never reused when an endpoint's state is evicted. A recreated record has no failure streak, so the first old in-flight failure can still count once; its remaining siblings cannot add further links to that streak. Timestamps still determine streak expiry and cooldown. Only a failure that was actually counted may trip the threshold: a sibling settling after the open window elapsed would otherwise start a fresh one off the existing count and push the half-open trial past the intended cooldown. - **A reset invalidates reports already in flight.** `reset()` bumps a per-host epoch instead of deleting the record, and a failure reported under an older epoch is discarded. Without that, the 30-second stragglers a user was waiting behind settle right after they press Retry and re-open the breaker underneath the very retry that cleared it. ### Trial ownership follows the request lifetime A half-open slot has no wall-clock expiry (#1439). A redirect chain gives each hop a separate timeout, and a body that keeps delivering bytes can exceed any fixed deadline without reaching its inactivity timeout. Neither may admit a second trial while the first request remains pending. The four request owners (Electron Xtream/Stalker IPC and web-backend `/xtream`/`/stalker`) acquire inside `try`, await the whole transport operation, report its outcome, and unconditionally release any remaining token in `finally`. `reportInconclusive` is the idempotent cleanup operation: it releases only the matching `trialId` and epoch and records no reachability evidence. This is the leak backstop even when debug logging, classification, or response formatting throws before an outcome report. Cleanup also runs while the environment kill switch is on, so switching it back off cannot revive a completed trial's slot. Desktop preference transitions invalidate the old guard and tokens as before. Cancellation releases after the awaited transport rejects, when it actually settles, rather than merely when an abort is requested. Xtream's AbortSignal continues through all Electron redirect hops. Stalker has no IPC cancellation signal, and closing a PWA client connection does not cancel the backend's outbound request; those slots remain owned until that transport settles. The transport's existing timeout handles inactivity. No heartbeat, polling timer, or larger trial deadline is needed. A future caller must preserve the `try`/`finally` contract; a custom transport that never settles needs cancellation at its own layer and must not be silently overlapped by a second trial. The slot still has an identity: delayed cleanup from a released owner must not free its replacement. A late failure still follows the ordinary streak and epoch rules; a real HTTP response still proves reachability and clears the record. Explicit reset, discovery success, preference transitions and bounded state eviction retain their existing rules for forgetting evidence. ## The fast-fail error is a renderer contract `HostConnectivityGuardError` is a real `Error`: Electron serializes a rejected plain object to `[object Object]`, which would destroy the renderer's classification. It carries no `status` property, because `getStalkerRequestErrorStatus` reads that field first. The message comes from `buildHostConnectivityFastFailMessage()` in `libs/shared/interfaces/src/lib/host-connectivity.util.ts`. It names the full endpoint, scheme included, so a user who imported the same panel over both HTTP and HTTPS can tell which one was skipped. Its wording is load-bearing — the Stalker renderer classifies transport failures purely from message text: | Must not contain | Otherwise | | ---------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------- | | `HTTP Error ` | reads as "endpoint absent, probe the next candidate"; 404/401/403 also fire lazy portal repair against a host we just declared dead | | `timed out`, `timeout of Nms`, `ETIMEDOUT` | discovery walks every candidate instead of aborting early | | `authorization`, `unauthorized`, `access denied`, `invalid token`, `auth failed`, `handshake failed` | fires lazy portal repair (a bare `authorization` matches) | What remains is the "connection-level failure" slot the renderer already has for ECONNREFUSED/ENOTFOUND: discovery stops probing and reports the host unreachable, `shouldAttemptRepair` returns false, and the message reaches error snackbars verbatim — which is why it reads like a sentence and names the endpoint. **The endpoint in that sentence is user data, so the marker outranks the heuristics.** A portal at `https://authorization.example` would otherwise make its own fast-fail message match the broad auth phrase set and send an unreachable host into lazy portal repair. `isStalkerAuthFailureMessage` and `isStalkerProbeTimeout` therefore both return false for a message `isHostConnectivityFastFailMessage` recognises, before their phrase matching runs. (`getStalkerRequestErrorStatus` needs no such guard: `HTTP Error ` contains a space, which a hostname cannot.) `stalker-portal-discovery.utils.spec.ts` pins all three properties for the bare and the IPC-wrapped form. ## Exemption: endpoint discovery `STALKER_REQUEST` accepts `skipConnectionGuard`, set **only** by `StalkerPortalDiscoveryService.probeContent`. Semantics: bypass the check, never count failures, **but still report successes.** Discovery walks several candidate paths on one host and expects most of them to fail; counting that would let it declare a slow-but-alive portal unreachable. Reporting successes is equally load-bearing: `confirmFullPortal` runs the full non-exempt authentication flow against auth-gated candidates, so without it two hung handshakes could open the breaker mid-discovery and the next authenticate would fast-fail into the "host unreachable" slot — abandoning a portal that works. "Success" here means any observed response, including one attached to a rejection: `validateStatus` lets 4xx through but rejects 5xx with `error.response` set, and a 5xx proves the origin answered just as well as a body does. That is why the exempt path reports through `reportGuardedHostFailure(token, error, { countFailures: false })` rather than skipping the report. The flag has to survive the PWA transport too, or discovery is exempt on the desktop and policed on the web. `PwaService.forwardStalkerRequest` forwards it as a `/stalker` control param and the route applies the same semantics — and, like `macAddress`/`token`/`serialNumber`, it is stripped from the query forwarded to the portal, because it is our control flag and not protocol content. ## Explicit reset `CONNECTIVITY_GUARD_RESET` (`{ url }`) is handled by `apps/electron-backend/src/app/events/connectivity-guard.events.ts`. One key derivation is enough: both `normalizeXtreamServerUrl` and `buildStalkerRequestUrl` rebuild their request URL from `URL.origin`, so the origin a request ends up using is always the origin of the URL stored on the playlist. **The rule: every user-driven retry or refresh that issues portal requests must reset the guard before its first request.** The failures that opened the breaker are usually the very ones the user is retrying, so a reset placed after the request — or missing — makes the affordance do nothing until the window expires. Automatic and first-load paths deliberately do NOT reset: only a user action means "contact this host now", and clearing evidence the guard just collected would defeat it. Call sites: - `retryContentInitialization` (`with-content.feature.ts`) — the Xtream content-gate Retry button. The reset is the **first** awaited statement, before the portal status check: a tripped guard fast-fails that check, its `unavailable` verdict returns early, and a reset placed any later would never run. - `StalkerPortalDiscoveryService.discover()` — one site covering import, the Edit dialog and lazy repair, which also guarantees a freshly edited address never inherits a refusal recorded for the previous one. - `retryContentPage` (`with-stalker-content.feature.ts`) — the Stalker grid tail's append retry. - `StalkerSearchComponent`'s search-page retry — the search results have their own append error and retry, separate from the catalog's. - `StalkerItvCacheService.refresh()` — the Live TV refresh button, on the same path that already clears the cache's own error cooldown. This also covers `refreshChannels()` in the live layout. - Both account-info dialogs' Retry buttons (`AccountInfoComponent.reload()` for Xtream, `StalkerAccountInfoComponent.reload()` for Stalker). Their automatic load on open goes through a private `load()` that does not reset. - The destructive Xtream refresh — before anything is deleted. It removes the cached catalog and then bootstraps a re-import whose status request an open guard would fast-fail, leaving the user with no catalog at all until the cooldown expires. The reset lives in `XtreamRefreshFlowService.runRefresh()` (`libs/playlist/shared/ui`), which owns the whole flow for both of its entry points: `PlaylistRefreshActionService.refreshXtream()` and `RecentPlaylistsComponent.refreshXtreamPlaylist()` (the Workspace sources page). Those two used to be independent near-duplicates, which is exactly why the second one was missed first time round; they now differ only in the `XtreamRefreshProgressReporter` they hand over, and a reporter cannot reach the reset. Keep it that way — a third entry point should pass a reporter, not copy the sequence. - `PortalStatusService.checkPortalStatusDetails` when `skipCache` is set — the user-initiated "Test Connection". Two retry paths deliberately have no reset: Xtream's `retryAppend()` is a no-op because in-memory appends cannot fail, and the guard's own half-open trial is not a user action. Where a retry clears a UI error flag, that flag is cleared **synchronously** before awaiting the reset — otherwise the retry branch stays re-enterable and the next `nearEnd` event fires a second retry. They all go through `resetHostConnectivityGuard()` (`libs/services/src/lib/host-connectivity-reset.ts`), which holds the one rule they share: the reset is best effort, because the guard only ever _delays_ a request and a failed reset must not block the action that asked for it. Both runtimes honour it, so neither keeps fast-failing an endpoint the user just asked to retry — in the PWA `PwaService` forwards it to the web backend's `POST /connectivity-guard/reset`, since the breaker lives in the backend process and a local reset would clear nothing. That route takes the raw provider URL rather than a registered `targetId`, because callers reset precisely when the address may have changed, which is before any target exists for it; it fetches nothing, reads only the origin, and never logs the URL. ## Interaction with VOD multi-source No exemption is needed. Multi-source resolution reaches `get_vod_info` over `XTREAM_REQUEST`, but `VodSourceResolverService.loadVodDetails` already catches that failure and falls back to its learned container cache or `null`, and `probeSource` maps `null` to the verdict `unknown` — which `VodSourceProbeCacheService` deliberately does not cache. A guard fast-fail therefore degrades to a retryable "could not check", exactly matching the module's contract that unreachable ≠ contacted-and-refused. The `STREAM_PROBE_URL` reachability half never rejects and is untouched. ## Tests - `libs/shared/host-health/src/lib/host-connectivity-guard.spec.ts` — the state machine, with an injected clock, long-lived trial ownership, late cleanup, and settlement while the environment kill switch is on. - `apps/electron-backend/src/app/util/host-connectivity-guard.spec.ts` — the main-process singleton and redirect attribution through it. - `apps/web-backend/src/app/web-backend-app.host-guard.spec.ts` — the proxy routes: the HTTP 200 refusal shape, that the outbound request really is skipped, that an open endpoint is refused before a DNS lookup is spent on it, redirect attribution, the discovery exemption, half-open, the reset endpoint, and that playlist/EPG downloads are never fast-failed. The per-route timeouts are asserted in `web-backend-app.spec.ts`, which pins the whole outbound request shape. - `apps/web/src/app/services/pwa.service.spec.ts` — that a fast-fail reaches the renderer with no numeric `status` and no `HTTP Error ` in its message, that the discovery bypass is forwarded, and that a reset calls the backend. - `apps/electron-backend/src/app/events/stalker.events.spec.ts` and `xtream.events.spec.ts` — trip, fast-fail without contacting axios, reset, exemption, long pending requests and redirect chains, cancellation, cleanup when debug reporting throws, and the absence of per-request log spam. - `libs/portal/stalker/data-access/src/lib/stalker-portal-discovery.utils.spec.ts` and `stalker-portal-repair.service.spec.ts` — the message contract.