refactor(storage): seams for server-backed playlist storage (#1896)

* docs(storage): server storage design and implementation plans

Design for keeping playlists and user data on the self-hosted server
(#305), revised against master, plus three implementation plans: storage
seams, the web-backend storage API, and the PWA server mode.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* refactor(database): share the playlist row codec

Move buildPlaylistRow, parseAppPlaylist and the upsert conflict helpers
from the Electron database operations into libs/shared/database
(@iptvnator/shared/database/app-playlist) so the self-hosted web backend
can write the same rows. The Electron modules re-export them; behavior is
unchanged.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* feat(database): read the data root from IPTVNATOR_DATA_DIR

Deployments that keep the database on a mounted volume, such as the
self-hosted web backend in Docker, can now set the data root. The E2E
override still wins; without either variable the root stays ~/.iptvnator.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* refactor(services): add the playlist storage backend resolver

PlaylistStorageBackend becomes the single place that decides whether
playlists persist through the Electron bridge or IndexedDB, and defines
the PlaylistDocumentStore contract a server store will implement.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* refactor(services): resolve playlist storage through one backend

PlaylistsService asks PlaylistStorageBackend for its document store
instead of reading window.electron and supportsSqlite directly. Paths that
only make sense on the desktop (legacy IndexedDB import, worker deletion
with progress, the favorites-channel query) check for the Electron kind
explicitly. Behavior is unchanged: the backend still resolves to the
Electron bridge or IndexedDB.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* refactor(xtream): store PWA user data through USER_DATA_STORAGE

PwaXtreamDataSource and PwaCollectionStorage read and write favorites,
recently viewed items, collection snapshots and playback positions through
the new USER_DATA_STORAGE token (default: localStorage) instead of calling
localStorage directly, so a deployment can substitute a server-synced
storage. Behavior is unchanged.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* docs(pwa): document the storage seams

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(database): bracket access in the shared playlist codec

libs/shared/database compiles with noPropertyAccessFromIndexSignature,
which the Electron backend did not enforce on the moved code.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* build(electron): alias the shared playlist codec in the worker bundle

The database worker is bundled by esbuild with an explicit alias table;
without the new entry, @iptvnator/shared/database/app-playlist resolved
under the library's index file.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(services): read localStorage lazily in USER_DATA_STORAGE

When the browser denies storage access, reading localStorage itself
throws. The default factory returned the global, so injecting the token
failed and PwaXtreamDataSource could not be constructed. The adapter now
reads the global inside each call, where the data source already guards
its reads.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* docs(database): describe /data as an example data root

The Docker image does not set IPTVNATOR_DATA_DIR yet; that arrives with
the server storage backend.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
4grayandClaude Opus 5.5 authored and GitHub committed 2026-10-11 20:03:58 +02:00
1 parent 633fde3e0e
commit 94570bc387
30 files changed
+8271 -450

No files matched your search

+7 -1
View File
@@ -33,7 +33,13 @@ import { content, categories, playlists, type Content } from '@iptvnator/shared/
## Database Location
The SQLite database is stored at: `~/.iptvnator/databases/iptvnator.db`
The SQLite database is stored at `<data root>/databases/iptvnator.db`. The data
root is resolved in this order:
1. `IPTVNATOR_E2E_DATA_DIR` (E2E runs only)
2. `IPTVNATOR_DATA_DIR` (an operator-selected data root for deployments
with a mounted volume, for example `/data`)
3. `~/.iptvnator`
## Upgrade Compatibility And Migrations
+1
View File
@@ -8,3 +8,4 @@ export * from './lib/schema';
export * from './lib/connection';
export * from './lib/path-utils';
export * from './lib/connection-observer';
export * from './lib/app-playlist';
@@ -0,0 +1,111 @@
import {
buildPlaylistRow,
parseAppPlaylist,
PLAYLIST_TYPES,
} from './app-playlist-codec';
import type { NewPlaylist, Playlist as PlaylistRow } from './schema';
function rowOf(document: Record<string, unknown>): NewPlaylist {
const row = buildPlaylistRow(document);
if (!row) {
throw new Error('Expected the document to produce a row');
}
return row;
}
/** What SQLite hands back for a row written from `row`. */
function storedRow(row: NewPlaylist, payload?: string | null): PlaylistRow {
const nullable = Object.fromEntries(
Object.entries(row).map(([key, value]) => [key, value ?? null])
);
return {
dateCreated: '2026-10-01 00:00:00',
...nullable,
payload: payload === undefined ? (row.payload ?? null) : payload,
} as PlaylistRow;
}
describe('app playlist codec', () => {
const m3uDocument = {
_id: 'm3u-1',
title: 'Family',
url: 'https://example.test/list.m3u',
count: 1,
importDate: '2026-10-01T00:00:00.000Z',
lastUsage: '2026-10-02T00:00:00.000Z',
favorites: ['https://example.test/a'],
recentlyViewed: [],
autoRefresh: true,
hiddenGroupTitles: ['Adult'],
playlist: {
header: { raw: '#EXTM3U' },
items: [
{
id: 'a',
url: 'https://example.test/a',
raw: '#EXTINF:-1,A',
},
],
},
};
it('round-trips an M3U document through a stored row', () => {
const row = rowOf(m3uDocument);
expect(row).toMatchObject({
id: 'm3u-1',
name: 'Family',
type: PLAYLIST_TYPES.M3U_URL,
url: 'https://example.test/list.m3u',
autoRefresh: true,
count: 1,
});
expect(parseAppPlaylist(storedRow(row))).toMatchObject({
_id: 'm3u-1',
title: 'Family',
favorites: ['https://example.test/a'],
hiddenGroupTitles: ['Adult'],
playlist: m3uDocument.playlist,
});
});
it('reads metadata from columns when the payload is not selected', () => {
const row = rowOf(m3uDocument);
const meta = parseAppPlaylist(storedRow(row, null));
expect(meta).toMatchObject({
_id: 'm3u-1',
title: 'Family',
count: 1,
favorites: ['https://example.test/a'],
url: 'https://example.test/list.m3u',
});
expect(meta['playlist']).toBeUndefined();
});
it('infers the provider type from the document', () => {
expect(
buildPlaylistRow({ _id: 'x', serverUrl: 'http://panel.test' })?.type
).toBe(PLAYLIST_TYPES.XTREAM);
expect(
buildPlaylistRow({
_id: 's',
macAddress: '00:1A:79:00:00:01',
portalUrl: 'http://portal.test/c',
})
).toMatchObject({
type: PLAYLIST_TYPES.STALKER,
url: 'http://portal.test/c',
});
expect(
buildPlaylistRow({ _id: 'f', filePath: '/tmp/list.m3u' })?.type
).toBe(PLAYLIST_TYPES.M3U_FILE);
expect(buildPlaylistRow({ _id: 't' })?.type).toBe(
PLAYLIST_TYPES.M3U_TEXT
);
});
it('refuses a document without an id', () => {
expect(buildPlaylistRow({ title: 'No id' })).toBeNull();
});
});
@@ -0,0 +1,263 @@
/**
* Row codec for the `playlists` table. The whole renderer `Playlist`
* document is stored as JSON in `payload`; the fields that metadata reads
* need are projected into columns. The Electron database worker and the
* self-hosted web backend both use this module, so both write the same rows.
*/
import * as schema from './schema';
export const PLAYLIST_TYPES = {
XTREAM: 'xtream',
STALKER: 'stalker',
M3U_FILE: 'm3u-file',
M3U_TEXT: 'm3u-text',
M3U_URL: 'm3u-url',
} as const;
export type AppPlaylistType =
(typeof PLAYLIST_TYPES)[keyof typeof PLAYLIST_TYPES];
const PLAYLIST_TYPE_VALUES = new Set<AppPlaylistType>(
Object.values(PLAYLIST_TYPES)
);
function getStringValue(value: unknown): string | undefined {
if (typeof value !== 'string') {
return undefined;
}
const normalized = value.trim();
return normalized.length > 0 ? normalized : undefined;
}
function getNumericValue(value: unknown): number | undefined {
if (typeof value === 'number' && Number.isFinite(value)) {
return value;
}
if (typeof value === 'string' && value.trim() !== '') {
const parsed = Number(value);
return Number.isFinite(parsed) ? parsed : undefined;
}
return undefined;
}
export function parseJsonValue<T>(
value: string | null | undefined,
fallback: T
): T {
if (!value) {
return fallback;
}
try {
return JSON.parse(value) as T;
} catch (error) {
console.warn('Failed to parse JSON value from DB:', error);
return fallback;
}
}
function getStringArrayValue(value: unknown): string[] {
if (!Array.isArray(value)) {
return [];
}
return value
.map((item) => (typeof item === 'string' ? item.trim() : ''))
.filter((item) => item.length > 0);
}
function inferPlaylistType(playlist: Record<string, unknown>): AppPlaylistType {
const explicitType = getStringValue(playlist['type']);
if (
explicitType &&
PLAYLIST_TYPE_VALUES.has(explicitType as AppPlaylistType)
) {
return explicitType as AppPlaylistType;
}
if (getStringValue(playlist['serverUrl'])) {
return PLAYLIST_TYPES.XTREAM;
}
if (getStringValue(playlist['macAddress'])) {
return PLAYLIST_TYPES.STALKER;
}
if (getStringValue(playlist['filePath'])) {
return PLAYLIST_TYPES.M3U_FILE;
}
if (getStringValue(playlist['url'])) {
return PLAYLIST_TYPES.M3U_URL;
}
return PLAYLIST_TYPES.M3U_TEXT;
}
export function buildPlaylistRow(
playlist: Record<string, unknown>
): schema.NewPlaylist | null {
const id =
getStringValue(playlist['_id']) ?? getStringValue(playlist['id']);
if (!id) {
return null;
}
const type = inferPlaylistType(playlist);
const portalUrl = getStringValue(playlist['portalUrl']);
const url = getStringValue(playlist['url']);
const nowIso = new Date().toISOString();
const updateDate = getNumericValue(playlist['updateDate']);
return {
id,
name:
getStringValue(playlist['title']) ??
getStringValue(playlist['name']) ??
id,
serverUrl: getStringValue(playlist['serverUrl']),
username: getStringValue(playlist['username']),
password: getStringValue(playlist['password']),
lastUpdated:
typeof updateDate === 'number'
? new Date(updateDate).toISOString()
: getStringValue(playlist['lastUpdated']),
type,
userAgent: getStringValue(playlist['userAgent']),
origin: getStringValue(playlist['origin']),
referrer: getStringValue(playlist['referrer']),
filePath: getStringValue(playlist['filePath']),
epgUrls:
playlist['epgUrls'] !== undefined
? JSON.stringify(getStringArrayValue(playlist['epgUrls']))
: undefined,
detectedEpgUrls:
playlist['detectedEpgUrls'] !== undefined
? JSON.stringify(
getStringArrayValue(playlist['detectedEpgUrls'])
)
: undefined,
manualEpgUrls:
playlist['manualEpgUrls'] !== undefined
? JSON.stringify(getStringArrayValue(playlist['manualEpgUrls']))
: undefined,
disabledEpgUrls:
playlist['disabledEpgUrls'] !== undefined
? JSON.stringify(
getStringArrayValue(playlist['disabledEpgUrls'])
)
: undefined,
autoRefresh: Boolean(playlist['autoRefresh']),
macAddress: getStringValue(playlist['macAddress']),
url: type === PLAYLIST_TYPES.STALKER ? (portalUrl ?? url) : url,
portalUrl,
count: getNumericValue(playlist['count']),
importDate: getStringValue(playlist['importDate']),
updateDate,
position: getNumericValue(playlist['position']),
favorites:
playlist['favorites'] !== undefined
? JSON.stringify(playlist['favorites'])
: undefined,
recentlyViewed:
playlist['recentlyViewed'] !== undefined
? JSON.stringify(playlist['recentlyViewed'])
: undefined,
payload: JSON.stringify(playlist),
lastUsage: getStringValue(playlist['lastUsage']) ?? nowIso,
};
}
export function getPlaylistItemCount(
playlist: Record<string, unknown>
): number | undefined {
try {
const value = playlist['playlist'];
if (typeof value !== 'object' || value === null) {
return undefined;
}
const items = Reflect.get(value, 'items');
return Array.isArray(items) ? items.length : undefined;
} catch {
return undefined;
}
}
export function parseAppPlaylist(
row: schema.Playlist
): Record<string, unknown> {
const payload = parseJsonValue<Record<string, unknown> | null>(
row.payload,
null
);
const base = payload && typeof payload === 'object' ? payload : {};
const favorites = parseJsonValue<unknown[]>(row.favorites, []);
const recentlyViewed = parseJsonValue<unknown[]>(row.recentlyViewed, []);
const epgUrls = getStringArrayValue(
row.epgUrls
? parseJsonValue<unknown[]>(row.epgUrls, [])
: base['epgUrls']
);
const detectedEpgUrls = getStringArrayValue(
row.detectedEpgUrls
? parseJsonValue<unknown[]>(row.detectedEpgUrls, [])
: (base['detectedEpgUrls'] ?? epgUrls)
);
const manualEpgUrls = getStringArrayValue(
row.manualEpgUrls
? parseJsonValue<unknown[]>(row.manualEpgUrls, [])
: base['manualEpgUrls']
);
const disabledEpgUrls = getStringArrayValue(
row.disabledEpgUrls
? parseJsonValue<unknown[]>(row.disabledEpgUrls, [])
: base['disabledEpgUrls']
);
const importDate =
row.importDate ?? row.dateCreated ?? new Date().toISOString();
const portalUrl =
row.portalUrl ?? (row.type === PLAYLIST_TYPES.STALKER ? row.url : null);
const updateDate =
row.updateDate ??
(row.lastUpdated ? new Date(row.lastUpdated).getTime() : undefined);
return {
...base,
_id: row.id,
title:
getStringValue(base['title']) ??
getStringValue(base['name']) ??
row.name,
count: row.count ?? getNumericValue(base['count']) ?? 0,
importDate: getStringValue(base['importDate']) ?? importDate,
lastUsage:
row.lastUsage ??
getStringValue(base['lastUsage']) ??
getStringValue(base['importDate']) ??
importDate,
favorites,
recentlyViewed,
autoRefresh: row.autoRefresh ?? Boolean(base['autoRefresh']),
url:
row.type === PLAYLIST_TYPES.M3U_URL
? (row.url ?? getStringValue(base['url']))
: getStringValue(base['url']),
filePath: row.filePath ?? getStringValue(base['filePath']),
epgUrls,
detectedEpgUrls,
manualEpgUrls,
disabledEpgUrls,
userAgent: row.userAgent ?? getStringValue(base['userAgent']),
referrer: row.referrer ?? getStringValue(base['referrer']),
origin: row.origin ?? getStringValue(base['origin']),
updateDate,
position: row.position ?? getNumericValue(base['position']),
serverUrl: row.serverUrl ?? getStringValue(base['serverUrl']),
username: row.username ?? getStringValue(base['username']),
password: row.password ?? getStringValue(base['password']),
macAddress: row.macAddress ?? getStringValue(base['macAddress']),
portalUrl: portalUrl ?? getStringValue(base['portalUrl']),
};
}
@@ -0,0 +1,81 @@
/**
* The `DO UPDATE` half of an app-playlist upsert and the payload clock
* reader, shared with the self-hosted web backend. The conditional writers
* that complete this contract stay in the Electron backend's
* `playlist-server-timezone.operations.ts` (issue #1562).
*/
import { sql, type SQL } from 'drizzle-orm';
import * as schema from './schema';
export type PlaylistConflictUpdate = {
[K in keyof schema.NewPlaylist]?: schema.NewPlaylist[K] | SQL;
};
const stored = schema.playlists;
/**
* The `DO UPDATE` half of an app-playlist upsert. A full upsert is built
* from a snapshot the caller read earlier; when that snapshot carries no
* panel clock while the stored row does — `setPlaylistServerTimezone`
* landed in between — the stored clock is carried over as long as the row
* still points at the same panel, so a favorites, recent-items or metadata
* write cannot hand a clockless payload back to the past. A snapshot that
* carries its own clock, or moves the source, wins as is. Nested CASE, not
* AND: SQLite may reorder AND terms, and the json_* readers raise on a
* malformed payload unless json_valid ran first.
*/
export function playlistConflictUpdate(
row: schema.NewPlaylist,
playlist: Record<string, unknown>
): PlaylistConflictUpdate {
if (readString(playlist['serverTimezone'])) {
return row;
}
const excludedPayload = sql.raw(`excluded.${stored.payload.name}`);
const payload = sql`CASE
WHEN json_valid(${stored.payload})
THEN CASE
WHEN json_type(${stored.payload}, '$.serverTimezone') = 'text'
AND ${stored.serverUrl} IS excluded.${sql.raw(stored.serverUrl.name)}
AND ${stored.username} IS excluded.${sql.raw(stored.username.name)}
AND ${stored.password} IS excluded.${sql.raw(stored.password.name)}
THEN json_set(
${excludedPayload},
'$.serverTimezone',
json_extract(${stored.payload}, '$.serverTimezone')
)
ELSE ${excludedPayload}
END
ELSE ${excludedPayload}
END`;
return { ...row, payload };
}
/**
* The clock stored in a raw row's payload, for `DB_GET_PLAYLIST`: the
* Xtream store seeds `currentPlaylist` from that read before (or without)
* the account-info check that learns the value.
*/
export function readPayloadServerTimezone(
payload: string | null | undefined
): string | undefined {
if (!payload) {
return undefined;
}
try {
const parsed: unknown = JSON.parse(payload);
return parsed && typeof parsed === 'object'
? readString((parsed as Record<string, unknown>)['serverTimezone'])
: undefined;
} catch {
return undefined;
}
}
function readString(value: unknown): string | undefined {
if (typeof value !== 'string') {
return undefined;
}
const normalized = value.trim();
return normalized.length > 0 ? normalized : undefined;
}
@@ -0,0 +1,2 @@
export * from './app-playlist-codec';
export * from './app-playlist-conflict';
@@ -9,6 +9,7 @@ import { existsSync, mkdtempSync, rmSync } from 'fs';
import { tmpdir } from 'os';
import { join } from 'path';
import {
IPTVNATOR_DATA_DIR_ENV,
IPTVNATOR_E2E_DATA_DIR_ENV,
getElectronConfigDirectory,
getElectronUserDataPath,
@@ -20,12 +21,15 @@ import {
describe('path-utils', () => {
let tempRoot: string;
let originalEnvValue: string | undefined;
let originalDataDirValue: string | undefined;
beforeEach(() => {
tempRoot = mkdtempSync(join(tmpdir(), 'iptvnator-path-utils-'));
homedirMock.mockReturnValue(join(tempRoot, 'home'));
originalEnvValue = process.env[IPTVNATOR_E2E_DATA_DIR_ENV];
delete process.env[IPTVNATOR_E2E_DATA_DIR_ENV];
originalDataDirValue = process.env[IPTVNATOR_DATA_DIR_ENV];
delete process.env[IPTVNATOR_DATA_DIR_ENV];
});
afterEach(() => {
@@ -34,9 +38,41 @@ describe('path-utils', () => {
} else {
process.env[IPTVNATOR_E2E_DATA_DIR_ENV] = originalEnvValue;
}
if (originalDataDirValue === undefined) {
delete process.env[IPTVNATOR_DATA_DIR_ENV];
} else {
process.env[IPTVNATOR_DATA_DIR_ENV] = originalDataDirValue;
}
rmSync(tempRoot, { force: true, recursive: true });
});
it('uses and creates IPTVNATOR_DATA_DIR as the data root', () => {
const dataDir = join(tempRoot, 'volume');
process.env[IPTVNATOR_DATA_DIR_ENV] = ` ${dataDir} `;
expect(getIptvnatorDataRoot()).toBe(dataDir);
expect(existsSync(dataDir)).toBe(true);
expect(getIptvnatorDatabasePath()).toBe(
join(dataDir, 'databases', 'iptvnator.db')
);
});
it('lets the E2E override win over IPTVNATOR_DATA_DIR', () => {
const e2eDir = join(tempRoot, 'e2e');
process.env[IPTVNATOR_E2E_DATA_DIR_ENV] = e2eDir;
process.env[IPTVNATOR_DATA_DIR_ENV] = join(tempRoot, 'volume');
expect(getIptvnatorDataRoot()).toBe(e2eDir);
});
it('ignores a blank IPTVNATOR_DATA_DIR', () => {
process.env[IPTVNATOR_DATA_DIR_ENV] = ' ';
expect(getIptvnatorDataRoot()).toBe(
join(tempRoot, 'home', '.iptvnator')
);
});
it('uses and creates the E2E data dir override when the env variable is set', () => {
const e2eDataDir = join(tempRoot, 'e2e-data');
process.env[IPTVNATOR_E2E_DATA_DIR_ENV] = e2eDataDir;
@@ -4,6 +4,12 @@ import { join } from 'path';
export const IPTVNATOR_E2E_DATA_DIR_ENV = 'IPTVNATOR_E2E_DATA_DIR';
/**
* Operator-selected data root for deployments that keep the database on a
* mounted volume, for example `/data`. The E2E override wins.
*/
export const IPTVNATOR_DATA_DIR_ENV = 'IPTVNATOR_DATA_DIR';
function ensureDirectory(dirPath: string): string {
if (!existsSync(dirPath)) {
mkdirSync(dirPath, { recursive: true });
@@ -19,6 +25,12 @@ export function getIptvnatorDataRoot(): string {
return ensureDirectory(e2eDataDir);
}
const dataDir = process.env[IPTVNATOR_DATA_DIR_ENV]?.trim();
if (dataDir) {
return ensureDirectory(dataDir);
}
return ensureDirectory(join(homedir(), '.iptvnator'));
}