iptvnator/libs/shared/database/README.md

Database

Shared database library for IPTVnator providing Drizzle ORM schema and connection utilities.

Usage

// Full access (electron-backend)
import { getDatabase, initDatabase } from '@iptvnator/shared/database';

// Read-only access (agent-backend)
import { getReadOnlyDatabase } from '@iptvnator/shared/database';

// Schema and types
import { content, categories, playlists, type Content } from '@iptvnator/shared/database';

Exports

Schema (schema.ts)

• Tables: playlists, categories, content, recentlyViewed, favorites
• Types: Playlist, Category, Content, RecentlyViewed, Favorite (and New* variants)

Connection (connection.ts)

• getDatabase(options?) - Full read-write access
• getReadOnlyDatabase() - Read-only access for agent queries
• initDatabase(options?) - Initialize with custom options
• closeDatabase() - Close connection
• getDatabasePath() - Get database file path

Connection observer (connection-observer.ts)

• setDatabaseConnectionObserver(observer | null) - Called by initDatabase with each connection it opens, before any statement runs on it. The Electron main process registers one only with IPTVNATOR_PERF_CAPTURE=1, to count main-thread SQL statements. The module has no runtime dependencies and is also importable as @iptvnator/shared/database/connection-observer.

Database Location

The SQLite database is stored at <data root>/databases/iptvnator.db. The data root is resolved in this order:

1. IPTVNATOR_E2E_DATA_DIR (E2E runs only)
2. IPTVNATOR_DATA_DIR (an operator-selected data root for deployments with a mounted volume, for example /data)
3. ~/.iptvnator

Upgrade Compatibility And Migrations

Users may skip releases. The application must apply every required migration in dependency order when opening an older database, preserving user data without requiring intermediate application installations or a database reset. This is the repository policy mirrored in AGENTS.md and CLAUDE.md.

src/lib/connection.ts owns initialization: createTables() creates missing schema objects, then runMigrations() applies column/index migrations and dedicated schema/data upgrades. One-off data migrations can record completion in app_state. Keep existing migration paths when adding new ones.

When changing initialization:

• Create required tables before migrating them. CREATE TABLE IF NOT EXISTS leaves existing columns unchanged.
• Add missing columns before creating indexes or triggers, or running queries, that depend on those columns. Put indexes depending on migrated columns in INDEX_MIGRATION_STATEMENTS, after COLUMN_MIGRATION_STATEMENTS, rather than in the earlier schema creation phase.
• Preserve existing data and make migrations safe on repeated startup. Record completion only after the associated migration succeeds.
• Exercise the actual initialization path with real SQLite and historical schema fixtures containing representative playlists, favorites, history, and playback positions. Verify both schema changes and preservation of those rows; mocked SQL calls alone do not establish upgrade compatibility.
• Include direct upgrades across skipped releases, the previous release, a fresh install, and repeated startup. If a release changes migration ordering, cover each distinct affected historical schema.

Two startup costs found on a 4 GB profile (3.9M catalog titles, 182 playlists) are guarded in runMigrations():

• deduplicateXtreamCache() returns at once when both unique indexes (src/lib/xtream-cache-unique-indexes.ts: categories_playlist_type_xtream_unique, content_category_type_xtream_unique) exist: they make duplicates impossible, and its two GROUP BY ... HAVING COUNT(*) > 1 scans cost a full pass over content on every launch otherwise (7.8 s cold on that profile). A profile whose unique index could not be created yet still gets the repair.
• payload must be the last column of playlists (PLAYLISTS_COLUMNS in src/lib/playlists-table.ts): SQLite reads a row's columns in order and walks the overflow pages of every column before the one it needs, so a column after an M3U payload costs the whole payload on each read (getAppPlaylistMetas took 4-22 s for 70 MB of payloads). Tables created before 0.19 got payload appended by ALTER TABLE with the EPG URL columns after it, and fresh installs up to 0.24 created last_usage after it; ensurePlaylistsPayloadLast() rebuilds such a table once (new table, copy, drop, rename, with foreign keys off) and is a no-op afterwards. It commits only if the tables referencing playlists hold no more foreign key violations than before; rows a legacy profile already orphaned do not block it. src/lib/playlists-table.spec.ts runs the rebuild and its rollback on real SQLite. The rebuild rewrites every payload, about 4 s on that profile; later starts open the database in about 40 ms instead of 5-8 s. Never append a column after payload; add it to PLAYLISTS_COLUMNS before payload and to COLUMN_MIGRATION_STATEMENTS, and the rebuild moves it on the next start.

The #1580 index-ordering fix is included in 0.24 through PR #1550. src/lib/connection-upgrades.spec.ts exercises initDatabase() with real SQLite under the Electron runtime, using fresh-install schema snapshots from tags 0.19–0.23 and a fresh current database. The epg_channel_id column is absent in 0.19, present from 0.20, and indexed from 0.23. Each case checks current Drizzle tables, columns/types, and named indexes/uniqueness, as well as preserved user rows, foreign keys, database integrity, index availability, and repeated startup; an existing EPG index must keep its definition and root page, and payload must end up as the last playlists column (the 0.19 snapshot reaches the current schema through the ALTER list, so it reproduces an upgraded profile's order). Snapshots live in src/lib/testing/fixtures/ and are independent of the current schema, so moving the index ahead of its column migration makes the 0.19 case fail again.

Run this coverage with pnpm nx test database --runInBand. The Electron UI and worker upgrade flow is also covered by pnpm nx run electron-backend-e2e:e2e-ci--src/legacy-playlist-migration.e2e.ts.