From 5febe28ebaca1505c052f9022443684d1258c22d Mon Sep 17 00:00:00 2001 From: 4gray <4gray@users.noreply.github.com> Date: Sun, 6 Sep 2026 08:59:28 +0200 Subject: [PATCH] fix(web-backend): validate and pin provider redirect hops (#1553) * fix(web-backend): validate and pin every provider redirect hop * fix(web-backend): separate provider metadata from connection authority --- .changes/web-backend-validated-redirects.md | 7 + AGENTS.md | 13 + CLAUDE.md | 13 + apps/web-backend/src/app/host-guard.ts | 24 +- .../src/app/provider-axios-transport.ts | 108 ++++++ apps/web-backend/src/app/provider-error.ts | 11 + .../src/app/provider-request-error.ts | 12 + apps/web-backend/src/app/provider-response.ts | 51 +++ .../src/app/provider-url-policy.spec.ts | 117 +++++++ .../src/app/provider-url-policy.ts | 140 ++++++++ .../src/app/testing/provider-test.key | 28 ++ .../src/app/testing/provider-test.pem | 20 ++ .../validated-http-client.integration.spec.ts | 323 ++++++++++++++++++ .../src/app/validated-http-client.spec.ts | 198 +++++++++++ .../src/app/validated-http-client.ts | 216 ++++++++++++ .../app/web-backend-app.host-guard.spec.ts | 79 ++--- .../src/app/web-backend-app.redirects.spec.ts | 238 +++++++++++++ .../src/app/web-backend-app.spec-helpers.ts | 32 +- .../src/app/web-backend-app.spec.ts | 2 +- apps/web-backend/src/app/web-backend-app.ts | 273 ++------------- apps/web-e2e/project.json | 1 + docs/architecture/host-connectivity-guard.md | 56 ++- docs/architecture/nx-workspace-boundaries.md | 6 + docs/architecture/pwa-self-hosted.md | 57 ++++ 24 files changed, 1694 insertions(+), 331 deletions(-) create mode 100644 .changes/web-backend-validated-redirects.md create mode 100644 apps/web-backend/src/app/provider-axios-transport.ts create mode 100644 apps/web-backend/src/app/provider-request-error.ts create mode 100644 apps/web-backend/src/app/provider-response.ts create mode 100644 apps/web-backend/src/app/provider-url-policy.spec.ts create mode 100644 apps/web-backend/src/app/provider-url-policy.ts create mode 100644 apps/web-backend/src/app/testing/provider-test.key create mode 100644 apps/web-backend/src/app/testing/provider-test.pem create mode 100644 apps/web-backend/src/app/validated-http-client.integration.spec.ts create mode 100644 apps/web-backend/src/app/validated-http-client.spec.ts create mode 100644 apps/web-backend/src/app/validated-http-client.ts create mode 100644 apps/web-backend/src/app/web-backend-app.redirects.spec.ts diff --git a/.changes/web-backend-validated-redirects.md b/.changes/web-backend-validated-redirects.md new file mode 100644 index 000000000..83c83a659 --- /dev/null +++ b/.changes/web-backend-validated-redirects.md @@ -0,0 +1,7 @@ +--- +type: fix +area: web-backend +issues: [1436] +--- + +The self-hosted backend now checks every provider redirect and pins connections to validated addresses, preventing redirects or DNS changes from bypassing private-network restrictions. Trusted LAN access remains available through the existing opt-in. diff --git a/AGENTS.md b/AGENTS.md index 4af88313b..fafb342ea 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -207,6 +207,19 @@ invalidation prevent late results from restoring removed programmes. Provider EPG is independent. See `docs/architecture/m3u-playlist-module.md` ("XMLTV source lifecycle"). +## Web Backend Provider Redirects + +All four provider proxy routes use `ValidatedHttpClient`: automatic redirects +are disabled, the initial URL and at most five redirect hops pass full URL/DNS +validation, and fresh agents pin each connection to that hop's validated IPs. +Host/SNI and TLS verification remain intact; outbound environment proxies are +disabled. Private-network opt-in applies to the chain. Cross-origin redirects +strip session headers; original query params are not replayed. One portal +admission owns the entire chain and final body, with explicit redirect evidence +preventing destination failures from penalizing the initial endpoint. Contracts: +`docs/architecture/pwa-self-hosted.md` and +`docs/architecture/host-connectivity-guard.md`. + ## Portal Connectivity Preference - Half-open trial slots follow the complete request lifetime with no elapsed-time diff --git a/CLAUDE.md b/CLAUDE.md index 22a023963..b7af8b73f 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -1692,6 +1692,19 @@ invalidation prevent late results from restoring removed programmes. Provider EPG is independent. See `docs/architecture/m3u-playlist-module.md` ("XMLTV source lifecycle"). +## Web Backend Provider Redirects + +All four provider proxy routes use `ValidatedHttpClient`: automatic redirects +are disabled, the initial URL and at most five redirect hops pass full URL/DNS +validation, and fresh agents pin each connection to that hop's validated IPs. +Host/SNI and TLS verification remain intact; outbound environment proxies are +disabled. Private-network opt-in applies to the chain. Cross-origin redirects +strip session headers; original query params are not replayed. One portal +admission owns the entire chain and final body, with explicit redirect evidence +preventing destination failures from penalizing the initial endpoint. Contracts: +`docs/architecture/pwa-self-hosted.md` and +`docs/architecture/host-connectivity-guard.md`. + ## Portal Connectivity Preference - Half-open trial slots follow the complete request lifetime with no elapsed-time diff --git a/apps/web-backend/src/app/host-guard.ts b/apps/web-backend/src/app/host-guard.ts index d4d710656..78ea157b1 100644 --- a/apps/web-backend/src/app/host-guard.ts +++ b/apps/web-backend/src/app/host-guard.ts @@ -13,9 +13,9 @@ * see `@iptvnator/shared/host-health`) fast-fails the rest once a host has * refused to answer twice in a row. * - * On the axios timeout semantics: with the default (follow-redirects) - * transport, `timeout` is NOT a wall-clock deadline for the whole response. It - * bounds the time to response headers, and then continues as the socket's + * On the axios timeout semantics: with axios 1.20's native HTTP + * transport (`maxRedirects: 0`), each hop's `timeout` is NOT a wall-clock + * deadline for the whole response. It bounds the time to response headers, and then continues as the socket's * inactivity timeout for the body. A large XMLTV or M3U download that keeps * delivering bytes is therefore never cut off mid-transfer — only a stalled one * is, which is exactly the case these numbers are meant to catch. @@ -29,6 +29,7 @@ import { portalEndpointKeyOf, } from '@iptvnator/shared/host-health'; import { buildHostConnectivityFastFailMessage } from '@iptvnator/shared/interfaces'; +import { ProviderRequestError } from './provider-request-error'; import type { NormalizedProviderError } from './provider-error'; /** @@ -176,8 +177,8 @@ export function resetProviderHost( * `countFailures: false` is for requests exempt from the guard (endpoint * discovery). Their failures are expected and must not count — but the report * must still happen, because an error that carries an HTTP response proves the - * endpoint answered. This route sets no `validateStatus`, so axios rejects every - * non-2xx WITH `error.response`; dropping those instead of reporting them is + * endpoint answered. The redirect wrapper accepts 3xx, but axios still rejects + * other non-2xx WITH `error.response`; dropping those instead of reporting them is * what would let the breaker open in the middle of discovery. */ export function reportProviderRequestFailure( @@ -190,6 +191,14 @@ export function reportProviderRequestFailure( return; } + const manualChain = error instanceof ProviderRequestError; + if (error instanceof ProviderRequestError) { + if (error.initialResponded) { + guard.reportSuccess(token); + return; + } + error = error.cause; + } const countFailures = options.countFailures ?? true; switch (classifyHostRequestFailure(error)) { case 'host-level': @@ -199,7 +208,10 @@ export function reportProviderRequestFailure( // other response — otherwise an ordinary timeout, a probe that was // redirected to a dead destination, and another ordinary timeout // still read as two consecutive failures. - if (failedAfterRedirect(error, token, options.requestUrl)) { + if ( + !manualChain && + failedAfterRedirect(error, token, options.requestUrl) + ) { guard.reportSuccess(token); break; } diff --git a/apps/web-backend/src/app/provider-axios-transport.ts b/apps/web-backend/src/app/provider-axios-transport.ts new file mode 100644 index 000000000..99b6aaae0 --- /dev/null +++ b/apps/web-backend/src/app/provider-axios-transport.ts @@ -0,0 +1,108 @@ +import axios from 'axios'; +import { + request as httpRequest, + ClientRequest, + IncomingMessage, + RequestOptions, +} from 'node:http'; +import { + request as httpsRequest, + Agent as HttpsAgent, + AgentOptions, +} from 'node:https'; +import { isIP } from 'node:net'; +import { checkServerIdentity } from 'node:tls'; +import type { + ProviderTransportOptions, + WebBackendHttpClient, + WebBackendHttpResponse, +} from './validated-http-client'; + +/** + * Keep HTTP authority metadata separate from socket destination selection. + * Only the hop's pinned agent can select an address; axios receives a fixed + * logical hostname, never a user-selected connection authority. Host and TLS + * identity still describe the validated provider, including IP certificates. + */ +export class ProviderAxiosTransport implements WebBackendHttpClient { + async get( + url: string, + options: ProviderTransportOptions = {} + ): Promise> { + const target = new URL(axios.getUri({ url, params: options.params })); + const secure = target.protocol === 'https:'; + const hostname = target.hostname.replace(/^\[|\]$/g, ''); + const port = Number(target.port || (secure ? 443 : 80)); + const root = secure + ? 'https://provider.invalid/' + : 'http://provider.invalid/'; + if (!options.httpAgent || !options.httpsAgent) { + throw new Error('Provider transport requires pinned agents'); + } + const agent = options.httpsAgent as HttpsAgent & { + options: AgentOptions; + }; + agent.options.servername = isIP(hostname) ? '' : hostname; + agent.options.checkServerIdentity = (_name, certificate) => + checkServerIdentity(hostname, certificate); + const response = await axios.get(root, { + ...options, + params: undefined, + transport: { + request: ( + requestOptions: RequestOptions, + callback: (response: IncomingMessage) => void + ) => { + // Path is HTTP metadata, never parsed as an authority. + // Even //host/path remains a path on the pinned connection. + const request = (secure ? httpsRequest : httpRequest)( + { + ...requestOptions, + hostname: 'provider.invalid', + port, + path: target.pathname + target.search, + }, + callback + ); + retainHeaderTimeout(request, options.timeout); + return request; + }, + }, + headers: { ...options.headers, Host: target.host }, + }); + return { + ...response, + headers: { + location: + typeof response.headers['location'] === 'string' + ? response.headers['location'] + : undefined, + }, + }; + } +} + +/** Axios identifies native transports by identity; a custom one needs this timer. */ +function retainHeaderTimeout(request: ClientRequest, timeout?: number): void { + if (!timeout) return; + const timer = setTimeout( + () => + request.destroy( + Object.assign( + new Error('Provider response headers timed out'), + { code: 'ECONNABORTED' } + ) + ), + timeout + ); + timer.unref(); + const clear = () => { + clearTimeout(timer); + request.removeListener('response', clear); + request.removeListener('error', clear); + request.removeListener('close', clear); + }; + request.once('response', clear); + request.once('error', clear); + request.once('close', clear); +} diff --git a/apps/web-backend/src/app/provider-error.ts b/apps/web-backend/src/app/provider-error.ts index 275e78706..9a64b9cd0 100644 --- a/apps/web-backend/src/app/provider-error.ts +++ b/apps/web-backend/src/app/provider-error.ts @@ -9,6 +9,9 @@ * routinely holds Xtream credentials. */ +import { ProviderRequestError } from './provider-request-error'; +import { providerUrlErrorBody } from './provider-url-policy'; + export interface ProviderError extends Error { readonly code?: unknown; readonly cause?: unknown; @@ -78,6 +81,10 @@ function visitErrorNode( export function normalizeProviderError( error: unknown ): NormalizedProviderError { + if (error instanceof ProviderRequestError) { + if (error.policyError) return providerUrlErrorBody(error.policyError); + return normalizeProviderError(error.cause); + } const providerError = error as ProviderError | null | undefined; const response = providerError?.response; if ( @@ -111,6 +118,10 @@ export function logProviderRequestFailure(options: { } function describeProviderFailure(error: unknown): string { + if (error instanceof ProviderRequestError) + return error.policyError + ? `URL policy ${error.policyError.status}` + : describeProviderFailure(error.cause); const providerError = error as ProviderError | null | undefined; const response = providerError?.response; if ( diff --git a/apps/web-backend/src/app/provider-request-error.ts b/apps/web-backend/src/app/provider-request-error.ts new file mode 100644 index 000000000..350e3c12b --- /dev/null +++ b/apps/web-backend/src/app/provider-request-error.ts @@ -0,0 +1,12 @@ +import type { ProviderUrlError } from './provider-url-policy'; + +/** Internal chain evidence, separate from the deliberately small public body. */ +export class ProviderRequestError extends Error { + constructor( + readonly initialResponded: boolean, + readonly cause: unknown, + readonly policyError?: ProviderUrlError + ) { + super('Provider request failed'); + } +} diff --git a/apps/web-backend/src/app/provider-response.ts b/apps/web-backend/src/app/provider-response.ts new file mode 100644 index 000000000..51400173f --- /dev/null +++ b/apps/web-backend/src/app/provider-response.ts @@ -0,0 +1,51 @@ +import { Socket } from 'node:net'; +import { Readable } from 'node:stream'; + +/** Redirect bodies are irrelevant, including invalid gzip and endless bodies. */ +export function discardProviderBody(data: unknown): void { + if (data instanceof Readable) data.destroy(); +} + +export function discardProviderErrorBody(error: unknown): void { + const response = (error as { response?: { data?: unknown } } | null) + ?.response; + discardProviderBody(response?.data); +} + +/** Match axios's buffered default and arraybuffer modes for the final hop. */ +export async function readProviderBody( + data: T, + arraybuffer: boolean, + timeout?: number, + socket?: Socket +): Promise { + if (!(data instanceof Readable)) return data; + const chunks: Buffer[] = []; + // Axios resolves stream responses at headers and then ignores its own + // request timeout callback. Keep the native socket's inactivity timer + // alive until the body finishes; arriving wire bytes reset it, even if + // decompression has not emitted another decoded chunk yet. + const onTimeout = () => + data.destroy( + Object.assign(new Error('Provider response body timed out'), { + code: 'ECONNABORTED', + }) + ); + if (timeout && socket) socket.setTimeout(timeout, onTimeout); + try { + for await (const chunk of data) chunks.push(Buffer.from(chunk)); + } finally { + if (timeout && socket) { + socket.removeListener('timeout', onTimeout); + socket.setTimeout(0); + } + } + const buffer = Buffer.concat(chunks); + if (arraybuffer) return buffer as T; + const text = buffer.toString('utf8').replace(/^\uFEFF/, ''); + try { + return JSON.parse(text) as T; + } catch { + return text as T; + } +} diff --git a/apps/web-backend/src/app/provider-url-policy.spec.ts b/apps/web-backend/src/app/provider-url-policy.spec.ts new file mode 100644 index 000000000..ef8b7c8dd --- /dev/null +++ b/apps/web-backend/src/app/provider-url-policy.spec.ts @@ -0,0 +1,117 @@ +import { validateProviderUrl } from './provider-url-policy'; + +const publicV4 = '93.184.216.34'; +const publicV6 = '2606:4700:4700::1111'; +const policy = { + allowPrivateNetworkTargets: false, + resolveHostname: async () => [publicV4], +}; +const blocked = [ + '0.0.0.0', + '10.1.2.3', + '100.64.0.1', + '127.0.0.1', + '169.254.169.254', + '172.16.0.1', + '192.168.0.1', + '192.0.2.1', + '198.18.0.1', + '198.51.100.1', + '203.0.113.1', + '224.0.0.1', + '255.255.255.255', + '::', + '::1', + 'fc00::1', + 'fd12::1', + 'fe80::1', + 'febf::1', + 'ff02::1', + '::ffff:127.0.0.1', + '::ffff:7f00:1', + '::ffff:a00:1', + '0:0:0:0:0:ffff:a9fe:a9fe', + '64:ff9b::a00:1', + '2001:db8::1', + '2001::1', + '2002:7f00:1::', + '3fff::1', +]; + +describe('provider URL policy', () => { + it.each(blocked)('rejects literal and DNS answer %s', async (address) => { + const url = `http://${address.includes(':') ? `[${address}]` : address}/`; + await expect(validateProviderUrl(url, policy)).resolves.toMatchObject({ + status: 400, + }); + await expect( + validateProviderUrl('https://provider.example', { + ...policy, + resolveHostname: async () => [publicV4, address], + }) + ).resolves.toMatchObject({ status: 400 }); + }); + it.each([ + 'ftp://host/', + 'file:///etc/passwd', + 'data:text/plain,hello', + 'http://user:secret@host', + 'invalid', + 'http://localhost.', + 'http://sub.localhost', + ])('rejects %s', async (url) => { + await expect(validateProviderUrl(url, policy)).resolves.toMatchObject({ + status: 400, + }); + }); + it.each([ + [], + ['bad-address'], + [publicV4, 'host.example'], + ['127.0.0.1%zone'], + ])( + 'rejects malformed DNS records %j even with LAN opt-in', + async (...addresses) => { + // Jest spreads array table rows; collect the records back into a list. + for (const allowPrivateNetworkTargets of [false, true]) { + await expect( + validateProviderUrl('https://provider.example', { + allowPrivateNetworkTargets, + resolveHostname: async () => addresses as string[], + }) + ).resolves.toMatchObject({ status: 400 }); + } + } + ); + it.each([publicV4, publicV6, '::ffff:5db8:d822'])( + 'accepts public address %s and retains the hostname', + async (address) => { + const result = await validateProviderUrl( + 'https://provider.example/path', + { + ...policy, + resolveHostname: async () => [address], + } + ); + expect(result).toEqual({ + url: new URL('https://provider.example/path'), + addresses: [address], + }); + } + ); + it('resolves and pins trusted LAN hosts while still rejecting schemes and credentials', async () => { + const trusted = { + allowPrivateNetworkTargets: true, + resolveHostname: async () => ['127.0.0.1', '::1'], + }; + await expect( + validateProviderUrl('https://lan.example', trusted) + ).resolves.toMatchObject({ addresses: ['127.0.0.1', '::1'] }); + await expect( + validateProviderUrl('https://user:secret@lan.example', trusted) + ).resolves.toMatchObject({ status: 400 }); + await expect( + validateProviderUrl('file:///tmp/test', trusted) + ).resolves.toMatchObject({ status: 400 }); + }); +}); diff --git a/apps/web-backend/src/app/provider-url-policy.ts b/apps/web-backend/src/app/provider-url-policy.ts new file mode 100644 index 000000000..862cf6536 --- /dev/null +++ b/apps/web-backend/src/app/provider-url-policy.ts @@ -0,0 +1,140 @@ +import { lookup } from 'node:dns/promises'; +import { BlockList, isIP } from 'node:net'; + +export interface ProviderUrlPolicy { + readonly allowPrivateNetworkTargets: boolean; + readonly resolveHostname: (hostname: string) => Promise; +} + +export interface ProviderUrlError { + readonly message: string; + readonly status: number; + readonly lookupError?: unknown; +} + +export interface ValidatedProviderTarget { + readonly url: URL; + readonly addresses: readonly string[]; +} + +export function providerUrlErrorBody(error: ProviderUrlError): { + message: string; + status: number; +} { + return { message: error.message, status: error.status }; +} + +export async function resolveHostname( + hostname: string +): Promise { + const records = await lookup(hostname, { all: true, verbatim: true }); + return records.map((record) => record.address); +} + +export async function validateProviderUrl( + rawUrl: string, + policy: ProviderUrlPolicy +): Promise { + let url: URL; + try { + url = new URL(rawUrl); + } catch { + return { message: 'Provider URL is not a valid URL', status: 400 }; + } + if (url.protocol !== 'http:' && url.protocol !== 'https:') { + return { + message: 'Only http and https provider URLs are supported', + status: 400, + }; + } + if (url.username || url.password) { + return { + message: 'Provider URL credentials are not supported', + status: 400, + }; + } + const hostname = url.hostname.replace(/^\[|\]$/g, '').toLowerCase(); + const local = hostname.replace(/\.$/, ''); + if ( + !policy.allowPrivateNetworkTargets && + (local === 'localhost' || + local.endsWith('.localhost') || + (isIP(hostname) !== 0 && !isPublicAddress(hostname))) + ) { + return privateAddressError(); + } + let addresses: readonly string[]; + try { + addresses = isIP(hostname) + ? [hostname] + : await policy.resolveHostname(hostname); + } catch (lookupError) { + return { + message: 'Provider URL host could not be resolved', + status: 400, + lookupError, + }; + } + // Validate every answer; never let a malformed record trigger a second DNS + // lookup in the transport. Even trusted LAN mode requires concrete IPs. + if (!addresses.length || addresses.some((address) => !isIP(address))) { + return { + message: 'Provider URL host could not be resolved', + status: 400, + }; + } + if ( + !policy.allowPrivateNetworkTargets && + addresses.some((address) => !isPublicAddress(address)) + ) { + return privateAddressError(); + } + return { url, addresses: [...addresses] }; +} + +function privateAddressError(): ProviderUrlError { + return { + message: 'Provider URL points to a private or local network address', + status: 400, + }; +} + +const blockedV4 = new BlockList(); +for (const [network, prefix] of [ + ['0.0.0.0', 8], + ['10.0.0.0', 8], + ['100.64.0.0', 10], + ['127.0.0.0', 8], + ['169.254.0.0', 16], + ['172.16.0.0', 12], + ['192.0.0.0', 24], + ['192.0.2.0', 24], + ['192.88.99.0', 24], + ['192.168.0.0', 16], + ['198.18.0.0', 15], + ['198.51.100.0', 24], + ['203.0.113.0', 24], + ['224.0.0.0', 3], +] as const) + blockedV4.addSubnet(network, prefix, 'ipv4'); + +const globalV6 = new BlockList(); +globalV6.addSubnet('2000::', 3, 'ipv6'); +const blockedV6 = new BlockList(); +for (const [network, prefix] of [ + ['2001::', 23], // Protocol assignments (including Teredo / benchmarking). + ['2001:db8::', 32], + ['2002::', 16], // Documentation / 6to4. + ['3fff::', 20], // Documentation. +] as const) + blockedV6.addSubnet(network, prefix, 'ipv6'); +const mappedV4 = new BlockList(); +mappedV4.addSubnet('::ffff:0:0', 96, 'ipv6'); + +function isPublicAddress(address: string): boolean { + if (isIP(address) === 4) return !blockedV4.check(address, 'ipv4'); + // Node BlockList handles IPv4-mapped IPv6 in both dotted and hex forms. + if (mappedV4.check(address, 'ipv6')) + return !blockedV4.check(address, 'ipv6'); + return globalV6.check(address, 'ipv6') && !blockedV6.check(address, 'ipv6'); +} diff --git a/apps/web-backend/src/app/testing/provider-test.key b/apps/web-backend/src/app/testing/provider-test.key new file mode 100644 index 000000000..64260c928 --- /dev/null +++ b/apps/web-backend/src/app/testing/provider-test.key @@ -0,0 +1,28 @@ +-----BEGIN PRIVATE KEY----- +MIIEvQIBADANBgkqhkiG9w0BAQEFAASCBKcwggSjAgEAAoIBAQC+U4KN4bcXoh+W +TMyRIBLi2jMWmqv8ZB5rcyiVvcqp4Z8aJOOuiSZjMJJxxsVwu6qKLZwm8KThYo4M +m5x5tXMOvUKhqxDoXhQcql5dAiydoFXCJD56cQnRpS/BT5PtqtSBG9VVo3v1YF13 +rKBF6ETcIiHFT7+jlSzPUP3lnyB7QL+yDFfaqX68TGXRkFG3thl3YMF+JhFILe7V +s8H2yZAiI3HuIeBUdTHOqnqVFW2vH1Hl4soXOsEQlQF5VyAFbCJ44duDfCe5uSbX +8lA+1VNJSRLKaQUdqvqgjF4UKmnk6a7xMAW0WTiC7RFoIUkH1WosprwEygYohhiF +XoG6QbKhAgMBAAECggEAA/H6NlOz9mbzbauo3+dAzPgF8BWDtCclJEgOUtBM16mo +ISQbnh4UsCCtIHOk2xngxp18a6g4Wr2uwR8mprU2rdsJew1vO8nbc96qNxZY82mD +7ZLPwrz+HZzleQXbxKTyY7y+dth9NNBrD5SB/AD9EG0asxrcl5j7hU6h/LUIONXN +GmOwEs7N7IsZgQtfItX8Zgo8+PUfQMp2QUawM4BAXwOSW3rpyioNmMQCa0Y/uKAf +8CKvNcP2XcbKucZNc9KHpNtaAjq7PDJ+7L/mSP3MGGNlYRjaasPiIemo2C6Pg1Pb +pr+yhwt28PFTqf2LLZcuivBtcPRlMmawcO9KFLjRewKBgQDm28ikYl8bqrROEgNQ +QL5rY/DzaaFjyEwvj+qWXWzy6cykQpw4vdLIfe3Kn9Evc4dU1o//RYnoqIWi7EJ0 +Ma6OBp4lMocHajklZdCG+ArQENcPcdRgVWGf6LZtEbPGaTk88mDOEMm50/FfnKGW +7FWsmpmcKqZduq2NX2BvzBDmtwKBgQDTDbTGIGelbe1Cl4ex/zBbcFA9CUm/4583 +1NMpfZNJaDbi9E3rOZK4ryCHAy4B9GQnmVW+A7ChzqMz2y9jKKM0RHlBXzvkroas +IQpRip11Vb/6ZXH3rM+7zgRYHi61+L7de2YbtCUfEOxQ+DcE5yNg1WzgRlYh0yfQ +SJ9f2XsZZwKBgEsVYn1sbShvbbMSkrdQR15gI+bXDSGJ7JVvhkmfWybqOZ+W9n5R +5rNEmclUD1ISjgpeuni44jCkVsp1cuudmPsiVd8dPuN/fdSW96peFA412+xvBjbK +rjS3GFYC8uhuIqqa3jdHKITi1NdW9wtCFF9N7PXovTEw3O9k/NV/lmOjAoGAWsDZ +DB0hFHS5gloQYozeKWOZTTWyPc5OR76/cmbqL7WdbGgrHUvreHjt3sCSRwrlClYY +FZYWnO1zJjhJHzV5QF91WJPv+DzH8jpe6oNVg//0hmKa6CqqRRKosY+A/ITS5gBK +/vyuvbYUOBkT54rQnrIHmEUGgpL+2sRvq9Kj6V8CgYEAxJ46AcgAW0SriYSn+oMX +KNMuAeFFS87fMEc+L3e/kH8ctWOGF8WAJew1v7TV+hHJvycMWqBbuvq7N1ArFBH4 +Hxq76pRoo6DMz5PxIVajSxj9+LuNWn9onuX9Ni0w2p9clH5nD1+Arnb2G91kZby8 +CqB+d4aWPHBq/0GQDp7xDR8= +-----END PRIVATE KEY----- diff --git a/apps/web-backend/src/app/testing/provider-test.pem b/apps/web-backend/src/app/testing/provider-test.pem new file mode 100644 index 000000000..a995b89d8 --- /dev/null +++ b/apps/web-backend/src/app/testing/provider-test.pem @@ -0,0 +1,20 @@ +-----BEGIN CERTIFICATE----- +MIIDNDCCAhygAwIBAgIUL/DZdKkHvo0op+ZSNbwEJ8/q98UwDQYJKoZIhvcNAQEL +BQAwGzEZMBcGA1UEAwwQcHJvdmlkZXIuZXhhbXBsZTAeFw0yNjA5MDYwNTU4Mjda +Fw0zNjA5MDMwNTU4MjdaMBsxGTAXBgNVBAMMEHByb3ZpZGVyLmV4YW1wbGUwggEi +MA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQC+U4KN4bcXoh+WTMyRIBLi2jMW +mqv8ZB5rcyiVvcqp4Z8aJOOuiSZjMJJxxsVwu6qKLZwm8KThYo4Mm5x5tXMOvUKh +qxDoXhQcql5dAiydoFXCJD56cQnRpS/BT5PtqtSBG9VVo3v1YF13rKBF6ETcIiHF +T7+jlSzPUP3lnyB7QL+yDFfaqX68TGXRkFG3thl3YMF+JhFILe7Vs8H2yZAiI3Hu +IeBUdTHOqnqVFW2vH1Hl4soXOsEQlQF5VyAFbCJ44duDfCe5uSbX8lA+1VNJSRLK +aQUdqvqgjF4UKmnk6a7xMAW0WTiC7RFoIUkH1WosprwEygYohhiFXoG6QbKhAgMB +AAGjcDBuMB0GA1UdDgQWBBSiXrRLEeZImTh1I0IjfqRQoHIUfzAfBgNVHSMEGDAW +gBSiXrRLEeZImTh1I0IjfqRQoHIUfzAPBgNVHRMBAf8EBTADAQH/MBsGA1UdEQQU +MBKCEHByb3ZpZGVyLmV4YW1wbGUwDQYJKoZIhvcNAQELBQADggEBAFtT545lqFbw +3pgHsnFI2kpXOso1WhjJa44cliFt0L2rg9JgZCih27ba1OZmr55Y6X2lG2xmQtqF +Gzm00pyC8EW/Yjwe2xcf0Tk2pGOTQ8MJlXhKOvwcxL6c6d/mc7Uvy+B/cwWbn9F9 +IeOgDiaPGTRWE/UgiVqq1ZARI3ExRN5G0QWejdw0Q3VYWh2L5lM9kKeRomudaX0T +YKQFuuuZji9/RQTN9ZRfjVG0imJOMfqi0Nrl4eva4kHP1FWzH7yR5ybjsuG9WiXx +6o7Ktfva+sRr//utJjluL8RvrxJ0d7Dd+BIKQQxBP3yL3YokweL1xzS2WdQM66QW +N+E360kOO9k= +-----END CERTIFICATE----- diff --git a/apps/web-backend/src/app/validated-http-client.integration.spec.ts b/apps/web-backend/src/app/validated-http-client.integration.spec.ts new file mode 100644 index 000000000..142257752 --- /dev/null +++ b/apps/web-backend/src/app/validated-http-client.integration.spec.ts @@ -0,0 +1,323 @@ +import { classifyHostRequestFailure } from '@iptvnator/shared/host-health'; +import { ProviderRequestError } from './provider-request-error'; +import { ProviderAxiosTransport } from './provider-axios-transport'; +import express from 'express'; +import { readFileSync } from 'node:fs'; +import { createServer, Agent as HttpsAgent } from 'node:https'; +import { Server } from 'node:http'; +import { AddressInfo } from 'node:net'; +import { TLSSocket } from 'node:tls'; +import { gzipSync } from 'node:zlib'; +import { + ValidatedHttpClient, + WebBackendHttpClient, +} from './validated-http-client'; +import { withServer } from './web-backend-app.spec-helpers'; + +const lan = { + allowPrivateNetworkTargets: true, + resolveHostname: async () => ['127.0.0.1'], +}; + +async function withListeningServer( + server: Server, + hostname: string, + run: (port: number) => Promise +): Promise { + await new Promise((resolve, reject) => { + server.once('error', reject); + server.listen(0, hostname, resolve); + }); + try { + return await run((server.address() as AddressInfo).port); + } finally { + server.closeAllConnections(); + await new Promise((resolve, reject) => + server.close((error) => (error ? reject(error) : resolve())) + ); + } +} + +describe('real axios validated transport', () => { + it('uses exactly the resolved address at connect time, preserves Host, and bypasses ambient proxy settings', async () => { + const proxy = express().use((_req, res) => + res.status(500).send('unexpected proxy') + ); + await withServer(proxy, async (proxyUrl) => { + const provider = express().use((req, res) => + res.json({ host: req.headers.host }) + ); + await withServer(provider, async (providerUrl) => { + const { port } = new URL(providerUrl); + const resolveHostname = jest + .fn() + .mockResolvedValueOnce(['127.0.0.1']) + .mockResolvedValue(['127.0.0.2']); + const previous = process.env['http_proxy']; + process.env['http_proxy'] = proxyUrl; + try { + const response = await new ValidatedHttpClient({ + ...lan, + resolveHostname, + }).get(`http://provider.example:${port}/`); + expect(response.data).toEqual({ + host: `provider.example:${port}`, + }); + expect(resolveHostname).toHaveBeenCalledTimes(1); + } finally { + if (previous === undefined) + delete process.env['http_proxy']; + else process.env['http_proxy'] = previous; + } + }); + }); + }); + it('uses the new validated DNS answer on each same-host redirect instead of a pooled socket', async () => { + const provider = express().use((_req, res) => res.redirect('/next')); + await withServer(provider, async (providerUrl) => { + const { port } = new URL(providerUrl); + const resolveHostname = jest + .fn() + .mockResolvedValueOnce(['127.0.0.1']) + .mockResolvedValueOnce(['127.0.0.2']); + await expect( + new ValidatedHttpClient({ ...lan, resolveHostname }).get( + `http://provider.example:${port}/`, + { timeout: 500 } + ) + ).rejects.toMatchObject({ + initialResponded: true, + cause: { + code: expect.stringMatching( + /^(ECONNREFUSED|ECONNABORTED)$/ + ), + }, + }); + expect(resolveHostname).toHaveBeenCalledTimes(2); + }); + }); + it.each(['invalid-gzip', 'unfinished'])( + 'discards the %s redirect body immediately', + async (kind) => { + const provider = express(); + provider.get('/start', (_req, res) => { + res.status(302).set('Location', '/final'); + if (kind === 'invalid-gzip') + res.set('Content-Encoding', 'gzip').end('not gzip'); + else res.write('a body that never ends'); + }); + provider.get('/final', (_req, res) => res.json({ ok: true })); + await withServer(provider, async (url) => { + await expect( + new ValidatedHttpClient(lan).get(`${url}/start`, { + timeout: 500, + }) + ).resolves.toMatchObject({ data: { ok: true } }); + }); + } + ); + it('preserves double-slash paths and escaped segments without changing connection authority', async () => { + const provider = express().use((req, res) => + res.json({ path: req.url, host: req.headers.host }) + ); + await withServer(provider, async (url) => { + const target = new URL(url); + await expect( + new ValidatedHttpClient(lan).get( + `${url}//other.example/a%2Fb?token=a%2Bb` + ) + ).resolves.toMatchObject({ + data: { + path: '//other.example/a%2Fb?token=a%2Bb', + host: target.host, + }, + }); + }); + }); + it('keeps query serialization and sends only Location query on a real redirect', async () => { + const requests: string[] = []; + const provider = express().use((req, res) => { + requests.push(req.url); + if (requests.length === 1) res.redirect('/final?ticket=issued'); + else res.send('done'); + }); + await withServer(provider, async (url) => { + await new ValidatedHttpClient(lan).get(`${url}/start?fixed=yes`, { + params: { password: 'a b+c' }, + }); + expect(requests).toEqual([ + '/start?fixed=yes&password=a+b%2Bc', + '/final?ticket=issued', + ]); + }); + }); + it('retains final arraybuffer, decompression, BOM text and JSON behavior', async () => { + const provider = express(); + provider.get('/binary', (_req, res) => + res.end(Buffer.from([0, 255, 128])) + ); + provider.get('/compressed', (_req, res) => + res.set('Content-Encoding', 'gzip').end(gzipSync('{"ok":true}')) + ); + provider.get('/text', (_req, res) => res.end('\uFEFFhello')); + await withServer(provider, async (url) => { + const client = new ValidatedHttpClient(lan); + expect( + ( + await client.get(`${url}/binary`, { + responseType: 'arraybuffer', + }) + ).data + ).toEqual(Buffer.from([0, 255, 128])); + expect((await client.get(`${url}/compressed`)).data).toEqual({ + ok: true, + }); + expect((await client.get(`${url}/text`)).data).toEqual('hello'); + }); + }); + it('cancels an unfinished final body and closes its connection', async () => { + const controller = new AbortController(); + let started!: () => void; + const bodyStarted = new Promise((resolve) => (started = resolve)); + const provider = express().use((_req, res) => { + res.write('unfinished'); + started(); + }); + await withServer(provider, async (url) => { + const pending = new ValidatedHttpClient(lan).get(url, { + signal: controller.signal, + }); + await bodyStarted; + controller.abort(); + await expect(pending).rejects.toMatchObject({ + cause: { code: 'ERR_CANCELED' }, + }); + }); + }); + it.each(['truncated', 'invalid-gzip', 'timeout'])( + 'retains response evidence for a %s final body', + async (kind) => { + const provider = express().use((_req, res) => { + if (kind === 'invalid-gzip') { + res.set('Content-Encoding', 'gzip').end('not gzip'); + return; + } + res.set('Content-Length', '100').write('short'); + if (kind === 'truncated') setTimeout(() => res.destroy(), 10); + }); + await withServer(provider, async (url) => { + const error = await new ValidatedHttpClient(lan) + .get(url, { timeout: 100 }) + .catch((error: unknown) => error); + expect(error).toBeInstanceOf(ProviderRequestError); + const failure = error as ProviderRequestError; + expect(failure.cause).toMatchObject({ + response: { status: 200, statusText: 'OK' }, + }); + expect(classifyHostRequestFailure(failure.cause)).toBe( + 'responded' + ); + }); + } + ); + it('times out before headers on the custom native transport', async () => { + const provider = express().use(() => { + /* Deliberately silent synthetic provider. */ + }); + await withServer(provider, async (url) => { + await expect( + new ValidatedHttpClient(lan).get(url, { timeout: 100 }) + ).rejects.toMatchObject({ + initialResponded: false, + cause: { code: 'ECONNABORTED' }, + }); + }); + }); + it('does not cap a healthy trickling body at the inactivity timeout', async () => { + const provider = express().use((_req, res) => { + let count = 0; + res.write('start'); + const interval = setInterval(() => { + res.write('x'); + if (++count === 8) { + clearInterval(interval); + res.end(); + } + }, 25); + res.on('close', () => clearInterval(interval)); + }); + await withServer(provider, async (url) => { + await expect( + new ValidatedHttpClient(lan).get(url, { timeout: 100 }) + ).resolves.toMatchObject({ data: 'startxxxxxxxx' }); + }); + }); + it('preserves TLS SNI and hostname verification with a pinned address', async () => { + // This key/certificate is exclusively a synthetic test fixture. + const cert = readFileSync(`${__dirname}/testing/provider-test.pem`); + const key = readFileSync(`${__dirname}/testing/provider-test.key`); + let requests = 0; + const server = createServer({ key, cert }, (req, res) => { + requests++; + res.setHeader('Content-Type', 'application/json'); + res.end( + JSON.stringify({ + host: req.headers.host, + sni: (req.socket as TLSSocket & { servername: string }) + .servername, + }) + ); + }); + const trustedTransport: WebBackendHttpClient = { + get: (url, options) => { + const agent = options?.httpsAgent as HttpsAgent & { + options: { ca?: Buffer }; + }; + // Supply the local test CA, retaining the production lookup, + // SNI and rejectUnauthorized defaults on the actual agent. + agent.options.ca = cert; + return new ProviderAxiosTransport().get(url, options); + }, + }; + await withListeningServer(server, '127.0.0.1', async (port) => { + await expect( + new ValidatedHttpClient(lan).get( + `https://provider.example:${port}` + ) + ).rejects.toMatchObject({ + cause: { code: 'DEPTH_ZERO_SELF_SIGNED_CERT' }, + }); + expect(requests).toBe(0); + const client = new ValidatedHttpClient(lan, trustedTransport); + await expect( + client.get(`https://provider.example:${port}`) + ).resolves.toMatchObject({ + data: { + host: `provider.example:${port}`, + sni: 'provider.example', + }, + }); + await expect( + client.get(`https://wrong.example:${port}`) + ).rejects.toMatchObject({ + cause: { code: 'ERR_TLS_CERT_ALTNAME_INVALID' }, + }); + expect(requests).toBe(1); + }); + }); + it('connects to a pinned IPv6 address and an IPv6 literal in trusted LAN mode', async () => { + const server = new Server((_req, res) => res.end('ipv6')); + await withListeningServer(server, '::1', async (port) => { + const client = new ValidatedHttpClient({ + ...lan, + resolveHostname: async () => ['::1'], + }); + expect( + (await client.get(`http://provider.example:${port}/`)).data + ).toBe('ipv6'); + expect((await client.get(`http://[::1]:${port}/`)).data).toBe( + 'ipv6' + ); + }); + }); +}); diff --git a/apps/web-backend/src/app/validated-http-client.spec.ts b/apps/web-backend/src/app/validated-http-client.spec.ts new file mode 100644 index 000000000..475977323 --- /dev/null +++ b/apps/web-backend/src/app/validated-http-client.spec.ts @@ -0,0 +1,198 @@ +import { Agent } from 'node:http'; +import { LookupFunction } from 'node:net'; +import { ValidatedHttpClient } from './validated-http-client'; +import { + resolvePublicHost, + StubHttpClient, +} from './web-backend-app.spec-helpers'; + +const policy = { + allowPrivateNetworkTargets: false, + resolveHostname: resolvePublicHost, +}; + +describe('validated HTTP redirect chain', () => { + it.each([301, 302, 303, 307, 308])( + 'follows relative Location for %s, without replaying original params', + async (status) => { + const transport = new StubHttpClient(); + transport.queueRedirect('../next?ticket=provider', status); + transport.queueResponse('done'); + const result = await new ValidatedHttpClient(policy, transport).get( + 'https://provider.example/base/start', + { + params: { username: 'demo', password: 'secret' }, + timeout: 1234, + } + ); + expect(result.data).toBe('done'); + expect(transport.requests[0].params).toEqual({ + username: 'demo', + password: 'secret', + }); + expect(transport.requests[1]).toMatchObject({ + url: 'https://provider.example/next?ticket=provider', + timeout: 1234, + }); + expect(transport.requests[1].params).toBeUndefined(); + } + ); + it('resolves fragment-only Location against the sent query and detects the cycle', async () => { + const transport = new StubHttpClient(); + transport.queueRedirect('#fragment'); + await expect( + new ValidatedHttpClient(policy, transport).get( + 'https://provider.example/start', + { + params: { token: 'secret' }, + } + ) + ).rejects.toMatchObject({ + policyError: { status: 502, message: 'Redirect cycle detected' }, + }); + expect(transport.requests).toHaveLength(1); + }); + it.each([ + ['https://provider.example/next', true], + ['https://provider.example:8443/next', false], + ['http://provider.example/next', false], + ['https://other.example/next', false], + ])( + 'scopes sensitive headers on redirect to %s', + async (location, retained) => { + const transport = new StubHttpClient(); + transport.queueRedirect(location as string); + transport.queueResponse('done'); + const headers = { + Authorization: 'Bearer secret', + cOoKiE: 'mac=secret', + 'Proxy-Authorization': 'secret', + sN: 'serial', + 'User-Agent': 'player', + }; + await new ValidatedHttpClient(policy, transport).get( + 'https://provider.example/start', + { headers } + ); + expect(transport.requests[1].headers).toEqual( + retained ? headers : { 'User-Agent': 'player' } + ); + expect(headers.Authorization).toBe('Bearer secret'); + } + ); + it.each([ + 'http://127.0.0.1/', + 'https://user:secret@provider.example/', + 'file:///tmp/test', + 'http://[invalid', + ])( + 'refuses unsafe Location %s without another transport call', + async (location) => { + const transport = new StubHttpClient(); + transport.queueRedirect(location); + await expect( + new ValidatedHttpClient(policy, transport).get( + 'https://provider.example/start' + ) + ).rejects.toMatchObject({ initialResponded: true }); + expect(transport.requests).toHaveLength(1); + } + ); + it('accepts five redirects and rejects a sixth before its destination', async () => { + for (const count of [5, 6]) { + const transport = new StubHttpClient(); + for (let i = 0; i < count; i++) transport.queueRedirect(`/hop${i}`); + transport.queueResponse('done'); + const request = new ValidatedHttpClient(policy, transport).get( + 'https://provider.example/start' + ); + if (count === 5) + await expect(request).resolves.toMatchObject({ data: 'done' }); + else + await expect(request).rejects.toMatchObject({ + policyError: { status: 502, message: 'Too many redirects' }, + }); + expect(transport.requests).toHaveLength(6); + } + }); + it('rejects a redirect without Location', async () => { + const transport = new StubHttpClient(); + transport.queueRedirect(''); + await expect( + new ValidatedHttpClient(policy, transport).get( + 'https://provider.example' + ) + ).rejects.toMatchObject({ policyError: { status: 502 } }); + }); + it('revalidates DNS on same-host hops and blocks a changed answer before connect', async () => { + const transport = new StubHttpClient(); + transport.queueRedirect('/next'); + const resolveHostname = jest + .fn() + .mockResolvedValueOnce(['93.184.216.34']) + .mockResolvedValueOnce(['127.0.0.1']); + await expect( + new ValidatedHttpClient( + { ...policy, resolveHostname }, + transport + ).get('https://provider.example/start') + ).rejects.toMatchObject({ + policyError: { status: 400 }, + initialResponded: true, + }); + expect(resolveHostname).toHaveBeenCalledTimes(2); + expect(transport.requests).toHaveLength(1); + }); + it('pins all validated IPv4/IPv6 records without a second DNS lookup', async () => { + const transport = new StubHttpClient(); + transport.queueResponse('done'); + const get = jest.spyOn(transport, 'get'); + const resolveHostname = jest + .fn() + .mockResolvedValue(['93.184.216.34', '2606:4700:4700::1111']); + await new ValidatedHttpClient( + { ...policy, resolveHostname }, + transport + ).get('https://provider.example/start'); + const options = get.mock.calls[0][1]; + if (!options) throw new Error('Missing transport options'); + expect(options).toMatchObject({ + maxRedirects: 0, + proxy: false, + adapter: 'http', + }); + const agent = options.httpsAgent as Agent & { + options: { lookup: LookupFunction; proxyEnv: object }; + }; + const lookup = agent.options.lookup as LookupFunction; + const callback = jest.fn(); + lookup('provider.example', { all: true }, callback); + expect(callback).toHaveBeenLastCalledWith(null, [ + { address: '93.184.216.34', family: 4 }, + { address: '2606:4700:4700::1111', family: 6 }, + ]); + lookup('provider.example', { family: 6 }, callback); + expect(callback).toHaveBeenLastCalledWith( + null, + '2606:4700:4700::1111', + 6 + ); + expect(resolveHostname).toHaveBeenCalledTimes(1); + expect(agent.options).toMatchObject({ proxyEnv: {} }); + }); + it('does not send a hop when canceled during its DNS validation', async () => { + const controller = new AbortController(); + const transport = new StubHttpClient(); + const resolveHostname = async () => { + controller.abort(); + return ['93.184.216.34']; + }; + await expect( + new ValidatedHttpClient( + { ...policy, resolveHostname }, + transport + ).get('https://provider.example', { signal: controller.signal }) + ).rejects.toMatchObject({ initialResponded: false }); + expect(transport.requests).toHaveLength(0); + }); +}); diff --git a/apps/web-backend/src/app/validated-http-client.ts b/apps/web-backend/src/app/validated-http-client.ts new file mode 100644 index 000000000..7fc4f4de2 --- /dev/null +++ b/apps/web-backend/src/app/validated-http-client.ts @@ -0,0 +1,216 @@ +import { ProviderAxiosTransport } from './provider-axios-transport'; +import { + discardProviderBody, + discardProviderErrorBody, + readProviderBody, +} from './provider-response'; +import axios, { AxiosRequestConfig } from 'axios'; +import { Agent as HttpAgent, ClientRequest } from 'node:http'; +import { Agent as HttpsAgent } from 'node:https'; +import { isIP, LookupFunction } from 'node:net'; +import { ProviderRequestError } from './provider-request-error'; +import { ProviderUrlPolicy, validateProviderUrl } from './provider-url-policy'; + +export interface WebBackendHttpGetOptions { + readonly headers?: Record; + readonly params?: Record; + readonly responseType?: 'arraybuffer'; + readonly timeout?: number; + readonly signal?: AbortSignal; +} + +export type ProviderTransportOptions = Omit< + WebBackendHttpGetOptions, + 'responseType' +> & + Pick< + AxiosRequestConfig, + | 'responseType' + | 'maxRedirects' + | 'validateStatus' + | 'httpAgent' + | 'httpsAgent' + | 'proxy' + | 'adapter' + >; + +export interface WebBackendHttpResponse { + readonly data: T; + readonly status: number; + readonly statusText?: string; + readonly request?: ClientRequest; + readonly headers: { readonly location?: string }; +} + +/** Injected transports must honor the same options and status contract as axios. */ +export interface WebBackendHttpClient { + get( + url: string, + options?: ProviderTransportOptions + ): Promise>; +} + +const REDIRECT_STATUSES = new Set([301, 302, 303, 307, 308]); +const SENSITIVE_HEADERS = new Set([ + 'authorization', + 'cookie', + 'proxy-authorization', + 'sn', +]); + +export class ValidatedHttpClient implements WebBackendHttpClient { + constructor( + private readonly policy: ProviderUrlPolicy, + private readonly transport: WebBackendHttpClient = new ProviderAxiosTransport() + ) {} + + async get( + rawUrl: string, + options: WebBackendHttpGetOptions = {} + ): Promise> { + let currentUrl = rawUrl; + let params = options.params; + let headers = options.headers; + let initialResponded = false; + const visited = new Set(); + try { + for (let redirects = 0; ; redirects++) { + options.signal?.throwIfAborted(); + const target = await validateProviderUrl( + currentUrl, + this.policy + ); + options.signal?.throwIfAborted(); + if ('message' in target) { + throw new ProviderRequestError( + initialResponded, + target.lookupError, + target + ); + } + // Match axios serialization once. Location is resolved against + // the URL actually sent; original params are never replayed. + const sentUrl = new URL( + axios.getUri({ url: target.url.href, params }) + ); + sentUrl.hash = ''; + if (visited.has(sentUrl.href)) + throw redirectError('Redirect cycle detected'); + visited.add(sentUrl.href); + const lookup = pinnedLookup(target.addresses); + // Fresh agents prevent socket-pool reuse across validations. + // Empty proxyEnv also disables Node's native env proxy support. + const agentOptions = { lookup, proxyEnv: {} }; + const httpAgent = new HttpAgent(agentOptions); + const httpsAgent = new HttpsAgent(agentOptions); + let response: WebBackendHttpResponse; + try { + response = await this.transport.get(target.url.href, { + ...options, + headers, + params, + adapter: 'http', + proxy: false, + maxRedirects: 0, + responseType: 'stream', + httpAgent, + httpsAgent, + validateStatus: (status) => + REDIRECT_STATUSES.has(status) || + (status >= 200 && status < 300), + }); + if (!REDIRECT_STATUSES.has(response.status)) { + try { + return { + ...response, + data: await readProviderBody( + response.data, + options.responseType === 'arraybuffer', + options.timeout, + response.request?.socket ?? undefined + ), + }; + } catch (error) { + if (axios.isCancel(error)) throw error; + // A broken/stalled body still proves the endpoint + // answered. Preserve buffered axios error semantics. + throw Object.assign( + new Error('Provider response body failed'), + { + cause: error, + response: { + status: response.status, + statusText: response.statusText, + }, + } + ); + } + } + initialResponded = true; + discardProviderBody(response.data); + } catch (error) { + discardProviderErrorBody(error); + throw error; + } finally { + httpAgent.destroy(); + httpsAgent.destroy(); + } + if (redirects >= 5) throw redirectError('Too many redirects'); + const location = response.headers.location; + if (!location) + throw redirectError( + 'Redirect response did not include a location' + ); + let nextUrl: URL; + try { + nextUrl = new URL(location, sentUrl); + } catch { + throw redirectError('Redirect location is not a valid URL'); + } + if (nextUrl.origin !== sentUrl.origin) { + headers = Object.fromEntries( + Object.entries(headers ?? {}).filter( + ([name]) => + !SENSITIVE_HEADERS.has(name.toLowerCase()) && + name.toLowerCase() !== 'host' + ) + ); + } + params = undefined; + currentUrl = nextUrl.href; + } + } catch (error) { + if (error instanceof ProviderRequestError) throw error; + throw new ProviderRequestError(initialResponded, error); + } + } +} + +function redirectError(message: string): ProviderRequestError { + return new ProviderRequestError(true, undefined, { message, status: 502 }); +} + +function pinnedLookup(addresses: readonly string[]): LookupFunction { + const records = addresses.map((address) => ({ + address, + family: isIP(address), + })); + return (_hostname, options, callback) => { + const eligible = options.family + ? records.filter((record) => record.family === options.family) + : records; + if (!eligible.length) { + callback( + Object.assign( + new Error('No validated address for the requested family'), + { code: 'ENOTFOUND' } + ), + [] + ); + } else if (options.all) { + callback(null, eligible); + } else { + callback(null, eligible[0].address, eligible[0].family); + } + }; +} diff --git a/apps/web-backend/src/app/web-backend-app.host-guard.spec.ts b/apps/web-backend/src/app/web-backend-app.host-guard.spec.ts index b01b3529b..91b9743ca 100644 --- a/apps/web-backend/src/app/web-backend-app.host-guard.spec.ts +++ b/apps/web-backend/src/app/web-backend-app.host-guard.spec.ts @@ -418,7 +418,7 @@ describe('web backend host connectivity guard', () => { // not consume the single trial the breaker allows. resolvable = false; const refusedByPolicy = await call(); - expect(refusedByPolicy.status).toBe(400); + expect(refusedByPolicy.status).toBe(200); resolvable = true; const trial = await call(); @@ -469,7 +469,7 @@ describe('web backend host connectivity guard', () => { ); const first = await call(); - expect(first.status).toBe(400); + expect(first.status).toBe(200); // The DNS error is internal: the client sees what it always saw. await expect(first.json()).resolves.toEqual({ message: 'Provider URL host could not be resolved', @@ -492,21 +492,11 @@ describe('web backend host connectivity guard', () => { }); it('does not fast-fail a provider whose redirect destination is dead', async () => { - // The shape this route actually produces, verified against axios - // 1.19.0: follow-redirects walks the chain inside one `get()`, so - // `config` still holds the URL we asked for and only - // `request._currentUrl` names the hop that failed. Reading `config.url` - // alone would compare the original URL with itself, find no redirect, - // and charge the dead destination to the provider that answered. - const redirectedFailure = () => - Object.assign(new Error('connect ECONNREFUSED'), { - code: 'ECONNREFUSED', - config: { url: 'http://xtream.example/player_api.php' }, - request: { _currentUrl: 'http://cdn.dead.example/stream' }, - }); const httpClient = new StubHttpClient(); - httpClient.queueNetworkError(redirectedFailure()); - httpClient.queueNetworkError(redirectedFailure()); + for (let i = 0; i < 2; i++) { + httpClient.queueRedirect('http://cdn.dead.example/stream'); + httpClient.queueNetworkError(hostLevelFailure()); + } httpClient.queueResponse({ user_info: { username: 'demo' } }); const { guard } = createTestGuard(); @@ -534,7 +524,7 @@ describe('web backend host connectivity guard', () => { action: 'get_account_info', payload: { user_info: { username: 'demo' } }, }); - expect(httpClient.requests).toHaveLength(3); + expect(httpClient.requests).toHaveLength(5); } ); }); @@ -689,31 +679,32 @@ describe('web backend host connectivity guard', () => { const started = new Promise((resolve) => { arrived = resolve; }); - const transport = jest - .spyOn(httpClient, 'get') - .mockImplementationOnce(async () => { - arrived(); - await pending; - if (outcome !== 'success') { - throw Object.assign( - hostLevelFailure( - outcome === 'cancel' - ? 'ERR_CANCELED' - : 'ETIMEDOUT' - ), - { - request: { - _currentUrl: - outcome === - 'redirect-failure' - ? 'http://cdn.example/slow' - : `http://portal.example/${route === 'xtream' ? 'player_api.php' : ''}`, - }, - } - ); - } - return { data: [] as never }; - }); + const transport = jest.spyOn(httpClient, 'get'); + if (outcome === 'redirect-failure') { + transport.mockImplementationOnce(async () => ({ + data: '' as never, + status: 302, + headers: { + location: 'http://cdn.example/slow', + }, + })); + } + transport.mockImplementationOnce(async () => { + arrived(); + await pending; + if (outcome !== 'success') { + throw hostLevelFailure( + outcome === 'cancel' + ? 'ERR_CANCELED' + : 'ETIMEDOUT' + ); + } + return { + data: [] as never, + status: 200, + headers: {}, + }; + }); const trial = call(); try { await started; @@ -729,7 +720,9 @@ describe('web backend host connectivity guard', () => { ).message ) ).toBe(true); - expect(transport).toHaveBeenCalledTimes(1); + expect(transport).toHaveBeenCalledTimes( + outcome === 'redirect-failure' ? 2 : 1 + ); } finally { settle(); await trial; diff --git a/apps/web-backend/src/app/web-backend-app.redirects.spec.ts b/apps/web-backend/src/app/web-backend-app.redirects.spec.ts new file mode 100644 index 000000000..df2882bfd --- /dev/null +++ b/apps/web-backend/src/app/web-backend-app.redirects.spec.ts @@ -0,0 +1,238 @@ +import { + HostConnectivityGuard, + OPEN_DURATION_MS, +} from '@iptvnator/shared/host-health'; +import { isHostConnectivityFastFailMessage } from '@iptvnator/shared/interfaces'; +import axios from 'axios'; +import express from 'express'; +import { createWebBackendApp, WebBackendHttpClient } from './web-backend-app'; +import { + registerProviderTarget, + resolvePublicHost, + StubHttpClient, + withServer, +} from './web-backend-app.spec-helpers'; + +describe('provider proxy redirect boundary', () => { + it.each(['/xtream', '/stalker', '/parse', '/parse-xml'])( + '%s refuses a private redirect before contacting its destination', + async (route) => { + let destinationRequests = 0; + const destination = express().get('/private', (_req, res) => { + destinationRequests++; + res.send('synthetic private response'); + }); + await withServer(destination, async (destinationUrl) => { + const provider = express().use((_req, res) => { + res.redirect(`${destinationUrl}/private`); + }); + await withServer(provider, async (providerUrl) => { + // Only the first public endpoint is mapped to a synthetic + // local provider. Axios and its redirect transport are real. + const httpClient: WebBackendHttpClient = { + get: (url, options) => + axios.get( + new URL(url).hostname === 'provider.example' + ? providerUrl + : url, + options + ), + }; + await withServer( + createWebBackendApp({ + httpClient, + resolveHostname: resolvePublicHost, + allowPrivateNetworkTargets: false, + }), + async (backend) => { + const id = await registerProviderTarget( + backend, + 'http://provider.example' + ); + const response = await fetch( + `${backend}${route}?targetId=${id}` + ); + const body = await response.json(); + expect(destinationRequests).toBe(0); + expect(response.status).toBe( + route.startsWith('/parse') ? 400 : 200 + ); + expect(body).toEqual({ + status: 400, + message: + 'Provider URL points to a private or local network address', + }); + } + ); + }); + }); + } + ); +}); + +describe('redirect routes and admission lifecycle', () => { + it.each(['/xtream', '/stalker', '/parse', '/parse-xml'])( + '%s follows an allowed local redirect and parses the final body', + async (route) => { + const provider = express().use((req, res) => { + if (req.path !== '/final') { + res.redirect('/final'); + return; + } + if (route === '/parse') + res.send( + '#EXTM3U\n#EXTINF:-1,Test\nhttps://example.com/test.ts' + ); + else if (route === '/parse-xml') + res.type('xml').send( + 'Test' + ); + else res.json({ ok: true }); + }); + await withServer(provider, async (url) => { + await withServer( + createWebBackendApp({ allowPrivateNetworkTargets: true }), + async (backend) => { + const id = await registerProviderTarget(backend, url); + const response = await fetch( + `${backend}${route}?targetId=${id}&action=test` + ); + const body = await response.json(); + expect(response.status).toBe(200); + if (route === '/parse') + expect(body).toMatchObject({ count: 1 }); + else if (route === '/parse-xml') + expect(body).toHaveProperty('channels'); + else + expect(body).toEqual({ + action: 'test', + payload: { ok: true }, + }); + } + ); + }); + } + ); + it.each(['/xtream', '/stalker', '/parse', '/parse-xml'])( + '%s rechecks a registered hostname before its initial connection', + async (route) => { + const transport = new StubHttpClient(); + const resolveHostname = jest + .fn() + .mockResolvedValueOnce(['93.184.216.34']) + .mockResolvedValue(['127.0.0.1']); + await withServer( + createWebBackendApp({ httpClient: transport, resolveHostname }), + async (backend) => { + const id = await registerProviderTarget( + backend, + 'https://provider.example' + ); + const response = await fetch( + `${backend}${route}?targetId=${id}` + ); + expect(response.status).toBe( + route.startsWith('/parse') ? 400 : 200 + ); + expect(await response.json()).toMatchObject({ + status: 400, + }); + expect(transport.requests).toHaveLength(0); + } + ); + } + ); + it.each(['/xtream', '/stalker'])( + '%s keeps query-only redirect failures off the initial endpoint record', + async (route) => { + const transport = new StubHttpClient(); + const networkFailure = () => + Object.assign( + new Error('secret http://user:password@provider.example'), + { code: 'ENOTFOUND' } + ); + transport.queueNetworkError(networkFailure()); + transport.queueRedirect('?next=1'); + transport.queueNetworkError(networkFailure()); + transport.queueNetworkError(networkFailure()); + transport.queueResponse({ ok: true }); + const guard = new HostConnectivityGuard(); + await withServer( + createWebBackendApp({ + httpClient: transport, + hostGuard: guard, + resolveHostname: resolvePublicHost, + }), + async (backend) => { + const id = await registerProviderTarget( + backend, + 'https://provider.example' + ); + const call = () => + fetch(`${backend}${route}?targetId=${id}`); + await call(); + const redirected = await call(); + expect(await redirected.text()).not.toContain('secret'); + await call(); + expect(await (await call()).json()).toMatchObject({ + payload: { ok: true }, + }); + expect(transport.requests).toHaveLength(5); + } + ); + } + ); + it.each(['/xtream', '/stalker'])( + '%s releases a half-open trial after redirect DNS refusal and counts initial DNS failures', + async (route) => { + let now = 1000; + const guard = new HostConnectivityGuard({ now: () => now }); + const transport = new StubHttpClient(); + let failDns = false; + const resolveHostname = async (hostname: string) => { + if (failDns || hostname === 'blocked.example') + throw Object.assign(new Error('secret transport details'), { + code: 'ENOTFOUND', + }); + return ['93.184.216.34']; + }; + await withServer( + createWebBackendApp({ + httpClient: transport, + hostGuard: guard, + resolveHostname, + }), + async (backend) => { + const id = await registerProviderTarget( + backend, + 'https://provider.example' + ); + const call = () => + fetch(`${backend}${route}?targetId=${id}`); + failDns = true; + await call(); + await call(); + const refused = (await (await call()).json()) as { + message: string; + }; + expect( + isHostConnectivityFastFailMessage(refused.message) + ).toBe(true); + expect(transport.requests).toHaveLength(0); + now += OPEN_DURATION_MS + 1; + failDns = false; + transport.queueRedirect('https://blocked.example/next'); + const trial = await call(); + expect(await trial.json()).toEqual({ + status: 400, + message: 'Provider URL host could not be resolved', + }); + transport.queueResponse({ ok: true }); + expect(await (await call()).json()).toMatchObject({ + payload: { ok: true }, + }); + } + ); + } + ); +}); diff --git a/apps/web-backend/src/app/web-backend-app.spec-helpers.ts b/apps/web-backend/src/app/web-backend-app.spec-helpers.ts index 2266cedaf..b4e882079 100644 --- a/apps/web-backend/src/app/web-backend-app.spec-helpers.ts +++ b/apps/web-backend/src/app/web-backend-app.spec-helpers.ts @@ -9,11 +9,11 @@ import { AddressInfo } from 'node:net'; import { Server } from 'node:http'; import { STALKER_MAG_USER_AGENT } from '@iptvnator/shared/interfaces'; +import { createWebBackendApp, WebBackendHttpClient } from './web-backend-app'; import { - createWebBackendApp, - WebBackendHttpClient, - WebBackendHttpGetOptions, -} from './web-backend-app'; + ProviderTransportOptions, + WebBackendHttpResponse, +} from './validated-http-client'; /** The transport-identity headers every portal-facing Stalker request carries. */ export const STALKER_IDENTITY_HEADERS = { @@ -41,12 +41,17 @@ export class StubHttpClient implements WebBackendHttpClient { readonly error?: Error; readonly status?: number; readonly statusText?: string; + readonly headers?: { location?: string }; }> = []; queueResponse(data: unknown): void { this.queuedResponses.push({ data }); } + queueRedirect(location: string, status = 302): void { + this.queuedResponses.push({ data: '', status, headers: { location } }); + } + queueFailure(status: number, statusText = 'Provider failure'): void { this.queuedResponses.push({ data: null, status, statusText }); } @@ -61,8 +66,8 @@ export class StubHttpClient implements WebBackendHttpClient { async get( url: string, - options: WebBackendHttpGetOptions = {} - ): Promise<{ data: T }> { + options: ProviderTransportOptions = {} + ): Promise> { this.requests.push({ headers: options.headers, params: options.params, @@ -90,7 +95,13 @@ export class StubHttpClient implements WebBackendHttpClient { throw error; } - if (response.status) { + if ( + response.status && + !( + options.validateStatus ?? + ((status) => status >= 200 && status < 300) + )(response.status) + ) { const error = new Error(response.statusText) as Error & { response: { status: number; statusText: string }; }; @@ -101,7 +112,11 @@ export class StubHttpClient implements WebBackendHttpClient { throw error; } - return { data: response.data as T }; + return { + data: response.data as T, + status: response.status ?? 200, + headers: response.headers ?? {}, + }; } } @@ -147,6 +162,7 @@ export async function withServer( const address = server.address() as AddressInfo; return await callback(`http://127.0.0.1:${address.port}`); } finally { + server.closeAllConnections(); await new Promise((resolve, reject) => { server.close((error) => (error ? reject(error) : resolve())); }); diff --git a/apps/web-backend/src/app/web-backend-app.spec.ts b/apps/web-backend/src/app/web-backend-app.spec.ts index 2a3080f82..f1086c944 100644 --- a/apps/web-backend/src/app/web-backend-app.spec.ts +++ b/apps/web-backend/src/app/web-backend-app.spec.ts @@ -893,7 +893,7 @@ https://stream.example/live.m3u8`); `${baseUrl}/xtream?targetId=${targetId}&action=get_account_info` ); - expect(response.status).toBe(400); + expect(response.status).toBe(200); await expect(response.json()).resolves.toEqual({ message: 'Provider URL points to a private or local network address', diff --git a/apps/web-backend/src/app/web-backend-app.ts b/apps/web-backend/src/app/web-backend-app.ts index 4ff17922a..f7529eea0 100644 --- a/apps/web-backend/src/app/web-backend-app.ts +++ b/apps/web-backend/src/app/web-backend-app.ts @@ -1,10 +1,7 @@ import cors from 'cors'; import express, { Express, Request, Response } from 'express'; import { createHash } from 'node:crypto'; -import { lookup } from 'node:dns/promises'; -import { isIP } from 'node:net'; import zlib from 'node:zlib'; -import axios from 'axios'; import epgParser from 'epg-parser'; import parser from 'iptv-playlist-parser'; import { @@ -33,19 +30,21 @@ import { ProviderError, } from './provider-error'; -export interface WebBackendHttpGetOptions { - readonly headers?: Record; - readonly params?: Record; - readonly responseType?: 'arraybuffer'; - readonly timeout?: number; -} - -export interface WebBackendHttpClient { - get( - url: string, - options?: WebBackendHttpGetOptions - ): Promise<{ data: T }>; -} +import { + ValidatedHttpClient, + WebBackendHttpClient, +} from './validated-http-client'; +import { ProviderRequestError } from './provider-request-error'; +import { + ProviderUrlPolicy, + providerUrlErrorBody, + resolveHostname, + validateProviderUrl, +} from './provider-url-policy'; +export type { + WebBackendHttpClient, + WebBackendHttpGetOptions, +} from './validated-http-client'; interface PlaylistParseError { readonly message: string; @@ -68,42 +67,12 @@ export interface WebBackendAppOptions { readonly runtimeBackendUrl?: string; } -interface ProviderUrlPolicy { - readonly allowPrivateNetworkTargets: boolean; - readonly resolveHostname: (hostname: string) => Promise; -} - -interface ProviderUrlError { - readonly message: string; - readonly status: number; - /** - * The DNS failure behind a "could not be resolved" refusal, when that is - * what this is. Internal only — {@link providerUrlErrorBody} strips it, - * because the client is told the same thing it always was. - * - * A name that does not resolve is evidence about reachability, exactly like - * the `ENOTFOUND` the transport would have raised a moment later. Without - * it, a host whose DNS died is refused by the policy on every request and - * the breaker never opens, so each call keeps paying for the lookup. - */ - readonly lookupError?: unknown; -} - -/** The client-facing half of a {@link ProviderUrlError}. */ -function providerUrlErrorBody(error: ProviderUrlError): { - message: string; - status: number; -} { - return { message: error.message, status: error.status }; -} - type ProviderTargetRegistry = Map; export function createWebBackendApp( options: WebBackendAppOptions = {} ): Express { const app = express(); - const httpClient = (options.httpClient ?? axios) as WebBackendHttpClient; const guid = options.guid ?? createGuid; const now = options.now ?? (() => new Date()); const hostGuard = @@ -125,6 +94,10 @@ export function createWebBackendApp( isPrivateNetworkProxyAllowed(), resolveHostname: options.resolveHostname ?? resolveHostname, }; + const httpClient = new ValidatedHttpClient( + providerUrlPolicy, + options.httpClient + ); const providerTargets: ProviderTargetRegistry = new Map(); const corsMiddleware = cors({ @@ -179,8 +152,8 @@ export function createWebBackendApp( return; } - const targetId = createProviderTargetId(result); - providerTargets.set(targetId, result); + const targetId = createProviderTargetId(result.url); + providerTargets.set(targetId, result.url); res.json({ targetId }); } ); @@ -303,38 +276,10 @@ export function createWebBackendApp( } guardToken = admission.token; - const providerUrlError = - await normalizeAndValidateXtreamProviderUrl( - url, - providerUrlPolicy - ); - if (providerUrlError) { - // Admitted, then abandoned before any request went out. A - // policy refusal — private address, bad scheme — says nothing - // about reachability, so it only hands the half-open slot back. - // A name that would not resolve is different: that IS the host - // failing to answer, and counting it is what lets the breaker - // stop paying for the same dead lookup on every request. - if (providerUrlError.lookupError !== undefined) { - reportProviderRequestFailure( - hostGuard, - guardToken, - providerUrlError.lookupError - ); - } else { - releaseProviderRequest(hostGuard, guardToken); - } - guardToken = null; - res.status(providerUrlError.status).json( - providerUrlErrorBody(providerUrlError) - ); - return; - } + url.href = normalizeXtreamServerUrl(url.href); requestUrl = appendPathSegment(url, 'player_api.php'); - // Provider URLs are validated by /provider-targets before they enter the registry. - // codeql[js/request-forgery] const response = await httpClient.get(requestUrl, { params: getProxyParams(req, ['targetId']), timeout: PROVIDER_REQUEST_TIMEOUT_MS.xtream, @@ -433,8 +378,6 @@ export function createWebBackendApp( guardToken = observeProviderRequest(hostGuard, requestUrl); } - // Provider URLs are validated by /provider-targets before they enter the registry. - // codeql[js/request-forgery] const response = await httpClient.get(requestUrl, { headers, // `create_link` gets the longer budget: the portal mints a @@ -492,78 +435,6 @@ function getRegisteredProviderUrl( return targetUrl; } -async function validateProviderUrl( - rawUrl: string, - policy: ProviderUrlPolicy -): Promise { - let url: URL; - try { - url = new URL(rawUrl); - } catch { - return { message: 'Provider URL is not a valid URL', status: 400 }; - } - - if (url.protocol !== 'http:' && url.protocol !== 'https:') { - return { - message: 'Only http and https provider URLs are supported', - status: 400, - }; - } - - if (url.username || url.password) { - return { - message: 'Provider URL credentials are not supported', - status: 400, - }; - } - - if (policy.allowPrivateNetworkTargets) { - return url; - } - - const hostname = normalizeHostname(url.hostname); - if (isLocalHostname(hostname) || isPrivateOrReservedIp(hostname)) { - return { - message: - 'Provider URL points to a private or local network address', - status: 400, - }; - } - - if (isIP(hostname) === 0) { - let addresses: readonly string[]; - try { - addresses = await policy.resolveHostname(hostname); - } catch (lookupError) { - return { - message: 'Provider URL host could not be resolved', - status: 400, - lookupError, - }; - } - - if ( - addresses.length === 0 || - addresses.some((address) => - isPrivateOrReservedIp(normalizeHostname(address)) - ) - ) { - return { - message: - 'Provider URL points to a private or local network address', - status: 400, - }; - } - } - - return url; -} - -async function resolveHostname(hostname: string): Promise { - const records = await lookup(hostname, { all: true, verbatim: true }); - return records.map((record) => record.address); -} - function createProviderTargetId(url: URL): string { return createHash('sha256').update(url.href).digest('hex'); } @@ -636,29 +507,6 @@ function appendPathSegment(url: URL, segment: string): string { return nextUrl.href; } -async function normalizeAndValidateXtreamProviderUrl( - url: URL, - policy: ProviderUrlPolicy -): Promise { - let normalizedUrl: URL; - try { - normalizedUrl = new URL(normalizeXtreamServerUrl(url.href)); - } catch { - return { message: 'Provider URL is not a valid URL', status: 400 }; - } - - const validatedUrl = await validateProviderUrl( - appendPathSegment(normalizedUrl, 'player_api.php'), - policy - ); - if ('message' in validatedUrl) { - return validatedUrl; - } - - url.href = normalizedUrl.href; - return null; -} - async function handlePlaylistParse(options: { readonly guid: () => string; readonly httpClient: WebBackendHttpClient; @@ -667,8 +515,6 @@ async function handlePlaylistParse(options: { readonly userAgent?: string; }): Promise | PlaylistParseError> { try { - // Provider URLs are validated by /provider-targets before playlist parsing. - // codeql[js/request-forgery] const response = await options.httpClient.get(options.url, { timeout: PROVIDER_REQUEST_TIMEOUT_MS.playlist, ...(options.userAgent @@ -689,14 +535,19 @@ async function handlePlaylistParse(options: { }; } catch (error) { logProviderRequestFailure({ error, route: '/parse', url: options.url }); - const providerError = error as ProviderError; + if (error instanceof ProviderRequestError && error.policyError) { + return providerUrlErrorBody(error.policyError); + } + const providerError = ( + error instanceof ProviderRequestError ? error.cause : error + ) as ProviderError; if (providerError?.response?.statusText !== undefined) { return { status: providerError.response.status ?? 500, message: providerError.response.statusText, }; } - const code = collectProviderErrorCodes(error)[0]; + const code = collectProviderErrorCodes(providerError)[0]; return { status: providerError?.response?.status ?? 500, message: code @@ -712,8 +563,6 @@ async function fetchEpgDataFromUrl( url: URL ): Promise { const href = url.href; - // Provider URLs are validated by /provider-targets before XMLTV parsing. - // codeql[js/request-forgery] const response = await httpClient.get(href, { timeout: PROVIDER_REQUEST_TIMEOUT_MS.epg, ...(url.pathname.endsWith('.gz') @@ -791,67 +640,3 @@ function getLastUrlSegment(value: string): string { function createGuid(): string { return Math.random().toString(36).slice(2); } - -function normalizeHostname(hostname: string): string { - return hostname.trim().replace(/^\[/, '').replace(/\]$/, '').toLowerCase(); -} - -function isLocalHostname(hostname: string): boolean { - return hostname === 'localhost' || hostname.endsWith('.localhost'); -} - -function isPrivateOrReservedIp(address: string): boolean { - const version = isIP(address); - if (version === 4) { - return isPrivateOrReservedIpv4(address); - } - - if (version === 6) { - return isPrivateOrReservedIpv6(address); - } - - return false; -} - -function isPrivateOrReservedIpv4(address: string): boolean { - const parts = address.split('.').map((part) => Number(part)); - if ( - parts.length !== 4 || - parts.some((part) => !Number.isInteger(part) || part < 0 || part > 255) - ) { - return true; - } - - const [first, second, third] = parts; - return ( - first === 0 || - first === 10 || - first === 127 || - (first === 100 && second >= 64 && second <= 127) || - (first === 169 && second === 254) || - (first === 172 && second >= 16 && second <= 31) || - (first === 192 && second === 168) || - (first === 192 && second === 0) || - (first === 192 && second === 0 && third === 2) || - (first === 198 && (second === 18 || second === 19)) || - (first === 198 && second === 51 && third === 100) || - (first === 203 && second === 0 && third === 113) || - first >= 224 - ); -} - -function isPrivateOrReservedIpv6(address: string): boolean { - const normalized = address.toLowerCase(); - if ( - normalized === '::' || - normalized === '::1' || - normalized.startsWith('fc') || - normalized.startsWith('fd') || - normalized.startsWith('fe80:') - ) { - return true; - } - - const mappedIpv4 = normalized.match(/::ffff:(\d+\.\d+\.\d+\.\d+)$/)?.[1]; - return mappedIpv4 ? isPrivateOrReservedIpv4(mappedIpv4) : false; -} diff --git a/apps/web-e2e/project.json b/apps/web-e2e/project.json index bba3cea4c..367cf8fb8 100644 --- a/apps/web-e2e/project.json +++ b/apps/web-e2e/project.json @@ -5,6 +5,7 @@ "sourceRoot": "apps/web-e2e/src", "implicitDependencies": [ "web", + "web-backend", "stalker-mock-server", "xtream-mock-server" ], diff --git a/docs/architecture/host-connectivity-guard.md b/docs/architecture/host-connectivity-guard.md index ecf2bd86b..956a8c5ce 100644 --- a/docs/architecture/host-connectivity-guard.md +++ b/docs/architecture/host-connectivity-guard.md @@ -47,11 +47,15 @@ so a silent host hung on OS-level TCP timeouts. Both runtimes now use the same budgets: Xtream 30 s, Stalker 15 s (30 s for `create_link`, which mints a stream URL before answering), playlist and XMLTV downloads 30 s. -Those numbers are safe for large downloads. On axios' default -(follow-redirects) transport `timeout` is **not** a wall-clock deadline for the +Those numbers are safe for large downloads. On axios 1.20's native HTTP +transport (`maxRedirects: 0`) each hop's `timeout` is **not** a wall-clock deadline for the whole response: it bounds the time to response headers and then continues as the socket's inactivity timeout for the body. A multi-megabyte XMLTV file that keeps -delivering bytes is never cut off mid-transfer — only a stalled one is. +delivering bytes is never cut off mid-transfer — only a stalled one is. The web +backend explicitly restores socket inactivity handling while reading the final +stream response, because axios stream mode stops its timeout handler at headers. +Truncated, malformed and timed-out final bodies retain their received response +status for reachability classification; cancellation retains its own semantics. ### Scope: portal calls only @@ -124,33 +128,27 @@ Merely declining to count it would leave an earlier direct failure standing, and a single later timeout would then fast-fail an endpoint that answered in between. Every caller passes the URL it asked for as the baseline. -**Where the failed hop is found depends on the transport, and both are in play.** +**Web backend records redirect evidence explicitly.** Its `ValidatedHttpClient` +disables automatic redirects and holds one admission for the entire chain. +`ProviderRequestError.initialResponded` becomes true only after receiving an +actual redirect response. A later transport or DNS failure, URL policy refusal, +invalid location, cycle or exhausted budget therefore clears the initial +endpoint's failure streak when the chain settles. This also handles redirects +that change only the query. It does not release the half-open slot early: the +route retains ownership through validation, all hops and the final body, and +releases in `finally` even when reporting throws. Before any redirect, initial +DNS failures carry their internal resolver cause into normal classification; +policy refusals remain inconclusive. Error bodies never serialize that cause. -| | Electron | Web backend | -| --- | --- | --- | -| Redirects | followed hop by hop (`maxRedirects: 0`), each its own request | followed inside one request by follow-redirects | -| Failed hop is in | `error.config.url` | `error.request._currentUrl` | - -`failedRequestUrlOf` reads `_currentUrl` first and falls back to `config.url`, -which is correct for both: a per-hop request exposes no `_currentUrl`, and on the -following transport `config` is built once and keeps the URL we asked for — so -reading `config.url` there would compare a URL with itself, find no redirect, and -charge a dead destination to the provider that answered. Anything added here must -work on both, because the same helper serves both. - -**The comparison is origin + path, not the whole URL.** A same-origin redirect -(`/player_api.php` → `/slow/player_api.php`) proves the endpoint answered just as -much as a cross-origin one, and charging it would fast-fail every OTHER call to a -portal that answers — so the path has to be part of it. The query must NOT be: -the web backend passes Xtream credentials through axios' `params`, so the sent -URL always carries a query the baseline does not, and comparing whole URLs made -every ordinary failure look like a redirect and stopped the breaker from ever -opening. What that gives up is a redirect that changes nothing but the query, -which is then counted as an ordinary failure — the safe direction. - -It requires positive evidence — anything unparseable or unknown counts the -failure as usual, because guessing "redirect" here would stop the guard from ever -tripping — and a failure that names no URL at all is still counted. +**Electron retains URL-based attribution.** `failedRequestUrlOf` reads +`error.request._currentUrl` first (for a following transport) and falls back to +`error.config.url` (for a native per-hop transport). The comparison is origin + +path, not the query: axios `params` can append credentials absent from the +caller's baseline. Unknown/unparseable URLs conservatively count as ordinary +failures, and query-only redirects cannot be distinguished by this fallback. +Wrapped web-backend requests use their explicit chain evidence and never infer +redirects from the transport's fixed logical hostname. The shared helper and +Electron behavior are unchanged by the web-backend fix. Known gap: the failing hop is not guarded either (it has no token of its own), so a permanently broken redirect chain keeps costing a full timeout. diff --git a/docs/architecture/nx-workspace-boundaries.md b/docs/architecture/nx-workspace-boundaries.md index d55cd9356..5dbb8d670 100644 --- a/docs/architecture/nx-workspace-boundaries.md +++ b/docs/architecture/nx-workspace-boundaries.md @@ -27,6 +27,12 @@ Do not invent a `test`, `build`, or `e2e` target because a similarly named project has one. Run affected lint/test/build targets that exist and the closest available E2E target for the changed behavior. +E2E applications must declare runtime dependencies even when they use HTTP +instead of TypeScript imports. `web-e2e` includes `web-backend` in +`implicitDependencies` so provider-proxy changes invalidate cached self-hosted +PWA tests; starting the backend through a `serve` dependency alone does not +make its source files inputs to the test hash. + ## Nx Dependency Updates Keep `nx` and every official `@nx/*` package on the same exact version. Run diff --git a/docs/architecture/pwa-self-hosted.md b/docs/architecture/pwa-self-hosted.md index d95ee24ab..eed5d6c0b 100644 --- a/docs/architecture/pwa-self-hosted.md +++ b/docs/architecture/pwa-self-hosted.md @@ -139,6 +139,63 @@ before any outbound request: - `IPTVNATOR_PROXY_ALLOW_PRIVATE_NETWORKS=1` explicitly enables trusted local/LAN targets for development, mock servers, or private deployments +### Provider redirects and connection policy + +`ValidatedHttpClient` applies the registration URL policy again before every +outbound request on `/xtream`, `/stalker`, `/parse`, and `/parse-xml`, including +the initial request. Axios automatic redirects are disabled (`maxRedirects: 0`); +301, 302, 303, 307 and 308 are followed manually, with at most five redirects. +Relative `Location` values resolve against the URL actually sent, including its +query. Missing/malformed locations, repeated URLs (ignoring fragments), and an +exhausted redirect budget fail without dispatching another request. Intermediate +bodies are destroyed at headers; malformed or endless redirect bodies do not +block following an otherwise valid location. Final bodies retain a native socket +inactivity timer: axios stream mode stops its own timeout handling at headers, +so `provider-response.ts` keeps that timer active until consumption finishes. +A body failure preserves received HTTP status evidence; cancellation remains +inconclusive before any redirect. + +Every DNS answer must be a valid permitted IP. Mixed public/private answers +fail closed. The strict policy rejects private/reserved IPv4, IPv4-mapped private +IPv6, and IPv6 outside global unicast `2000::/3`, plus protocol-assignment, +documentation and 6to4 ranges within it. These deliberately conservative ranges +follow the [IANA IPv6 address space](https://www.iana.org/assignments/ipv6-address-space/) +and [special-purpose registry](https://www.iana.org/assignments/iana-ipv6-special-registry/). +The LAN opt-in applies to the whole +chain; it still requires HTTP(S), no URL userinfo and concrete DNS addresses. + +Fresh HTTP/HTTPS agents pin socket lookup to the exact validated IPv4/IPv6 +answer set for that hop. No second DNS query or pooled connection may substitute +an unvalidated address. The default axios transport uses a fixed logical hostname (`provider.invalid`) +so its connection authority cannot come from user-controlled URL metadata; +the pinned lookup alone selects an IP. Its native request-options adapter sets +`path` separately, so even `//host/path` cannot replace the connection authority. +It also owns the deadline from dispatch through response headers, since axios's +built-in connection timer only covers its own native transport objects. The original provider host and port are +explicitly preserved in Host, TLS SNI and certificate identity checks (literal +IPs omit SNI but still verify the original IP). Axios proxies and Node environment proxies +are disabled for these requests so a proxy cannot independently resolve the +origin. Deployments that require an outbound HTTP proxy must use a different +network arrangement; setting `HTTP_PROXY`/`HTTPS_PROXY` does not route provider +requests through it. This guarantee concerns address selection in the Node +transport, not routing/NAT performed outside the process. Private-network +opt-in intentionally relaxes address restrictions. + +Original axios `params` are applied only to the initial request. Subsequent +queries come from `Location` resolution, without appending the original +credentials again. Authorization, Cookie, Proxy-Authorization and Stalker SN +headers are retained only on the same origin; changes of host, port or scheme +(including HTTPS downgrades) strip them for the rest of the chain. User-Agent +and other non-secret protocol headers survive. Providers can explicitly put +query values in `Location`; the backend does not rewrite provider-issued URLs. + +Policy errors contain fixed messages/statuses without URLs, DNS exceptions or +transport objects. Portal routes keep HTTP 200 with a `{ message, status }` +error envelope; playlist/XMLTV routes use the actual error status. Registration +continues to use real HTTP error statuses. See +[host connectivity guard](host-connectivity-guard.md) for chain ownership and +failure attribution. + Do not disable TLS certificate validation in the backend proxy. For private certificate authorities, configure Node with `NODE_EXTRA_CA_CERTS`.