fix(portals): fold diacritics in the title index

Cross-playlist matching compares normalized titles ("Amélie" -> "amelie")
against an index built from the raw title, and the trigram tokenizer does not
fold diacritics by default. Every accented title was therefore invisible to
the FTS path: two identical `Amélie` entries produced no candidates at all.
That is the broadest of the Unicode gaps review found, and it predates the
short-title work.

The tokenizer is fixed at CREATE time, so existing databases recreate and
rebuild the index once behind a migration marker. `remove_diacritics` needs
SQLite 3.45+, so support is probed on a temp table first: an older runtime
keeps its working index untouched and the migration is not recorded as done,
leaving a later version free to upgrade it.

Case folding for non-ASCII remains impossible in stock SQLite — "ОН" cannot
find "Он" by any available predicate — and is documented as the known limit
rather than patched around again.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
4grayandClaude Opus 5 committed 2026-07-29 08:15:30 +02:00
1 parent a0eb68196d
commit 0b46aadee0
3 files changed
+209 -1

No files matched your search

+20
View File
@@ -481,6 +481,26 @@ both sources running and Stop owning only the newer one.
## Short titles and Unicode
The title index folds diacritics. `content_title_fts` is created with
`tokenize='trigram remove_diacritics 1'`, because matching compares NORMALIZED
titles ("Amélie" → "amelie") against an index built from the raw one — without
folding, every accented title was invisible to the FTS path, and two identical
`Amélie` entries produced no candidates at all. The tokenizer is fixed at
CREATE time, so existing databases are recreated and rebuilt once behind
`migration:content-title-fts-remove-diacritics:v1`.
`remove_diacritics` needs SQLite 3.45+. The migration probes support on a temp
table first and does nothing when the runtime rejects it, leaving the working
index in place — and does NOT record itself as done, so a later app version
shipping a newer SQLite upgrades then.
**Case folding for non-ASCII is still not possible.** `LOWER()`, GLOB classes
and the trigram tokenizer all fold ASCII only, so a Cyrillic title stored with
different capitalisation in two playlists ("ОН" vs "Он") cannot be matched by
any predicate available in stock SQLite. Closing that needs a stored
normalized-title column, which is deliberately out of scope here.
SQLite's `LOWER()` and GLOB character classes are ASCII-only, so a short
non-ASCII title could not be folded or word-bounded and simply never matched —
the film stayed absent from the chip. The ASCII/Unicode branch is decided from the RAW token, not the normalized