mirror of
https://github.com/4gray/iptvnator.git
synced 2026-10-16 14:16:16 -08:00
fix(portals): fold diacritics in the title index
Cross-playlist matching compares normalized titles ("Amélie" -> "amelie")
against an index built from the raw title, and the trigram tokenizer does not
fold diacritics by default. Every accented title was therefore invisible to
the FTS path: two identical `Amélie` entries produced no candidates at all.
That is the broadest of the Unicode gaps review found, and it predates the
short-title work.
The tokenizer is fixed at CREATE time, so existing databases recreate and
rebuild the index once behind a migration marker. `remove_diacritics` needs
SQLite 3.45+, so support is probed on a temp table first: an older runtime
keeps its working index untouched and the migration is not recorded as done,
leaving a later version free to upgrade it.
Case folding for non-ASCII remains impossible in stock SQLite — "ОН" cannot
find "Он" by any available predicate — and is documented as the known limit
rather than patched around again.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
1 parent
a0eb68196d
commit
0b46aadee0
3 files changed
+209
-1
No files matched your search
@@ -481,6 +481,26 @@ both sources running and Stop owning only the newer one.
|
||||
|
||||
## Short titles and Unicode
|
||||
|
||||
The title index folds diacritics. `content_title_fts` is created with
|
||||
`tokenize='trigram remove_diacritics 1'`, because matching compares NORMALIZED
|
||||
titles ("Amélie" → "amelie") against an index built from the raw one — without
|
||||
folding, every accented title was invisible to the FTS path, and two identical
|
||||
`Amélie` entries produced no candidates at all. The tokenizer is fixed at
|
||||
CREATE time, so existing databases are recreated and rebuilt once behind
|
||||
`migration:content-title-fts-remove-diacritics:v1`.
|
||||
|
||||
`remove_diacritics` needs SQLite 3.45+. The migration probes support on a temp
|
||||
table first and does nothing when the runtime rejects it, leaving the working
|
||||
index in place — and does NOT record itself as done, so a later app version
|
||||
shipping a newer SQLite upgrades then.
|
||||
|
||||
**Case folding for non-ASCII is still not possible.** `LOWER()`, GLOB classes
|
||||
and the trigram tokenizer all fold ASCII only, so a Cyrillic title stored with
|
||||
different capitalisation in two playlists ("ОН" vs "Он") cannot be matched by
|
||||
any predicate available in stock SQLite. Closing that needs a stored
|
||||
normalized-title column, which is deliberately out of scope here.
|
||||
|
||||
|
||||
SQLite's `LOWER()` and GLOB character classes are ASCII-only, so a short
|
||||
non-ASCII title could not be folded or word-bounded and simply never matched —
|
||||
the film stayed absent from the chip. The ASCII/Unicode branch is decided from the RAW token, not the normalized
|
||||
|
||||
Reference in new issue
Block a user