fix(portals): fold non-ASCII case in the scan, and read years as tags

I was wrong about SQLite twice over, and both errors cost matches.

GLOB character classes are NOT ASCII-only. `patternCompare` reads them as
UTF-8 code points, so `'Он' GLOB '*[Оо][Нн]*'` is true — only `LOWER()` is
ASCII-only. The scan tier now folds the case in JavaScript, where Unicode
case mapping is real, and hands SQLite one class per character. A short
Cyrillic or Greek title stored in a different case is found instead of
being silently absent from the Sources chip. The builder returns `null`,
leaving the substring tests as the whole answer, for a token holding a
GLOB metacharacter (GLOB has no escape character) or a case mapping that
changes length. The FTS tier is untouched and still cannot fold — that
needs a stored normalized-title column.

The movie's own year came from `extractYear`, which reads a year from
anywhere in the title. That is right where a year is a search hint, wrong
where it is an identity: `2001: A Space Odyssey` was treated as a 2001
film, so every genuine 1968 copy failed the year gate and the movie had
no alternatives at all — and its pin key moved the moment enrichment
supplied the real year. `releaseTagYear` accepts only bracketed and
trailing forms; the repo's own TRAILING_YEAR_PATTERN already documented
this exact hazard.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
4grayandClaude Opus 5 committed 2026-07-29 19:48:25 +02:00
1 parent bbdc64b9dc
commit ddb299e237
9 files changed
+305 -33

No files matched your search

+31 -15
View File
@@ -285,6 +285,16 @@ year, ranked *above* every fuzzy match. Discovery reads a bracketed year out of
the raw title first, and when both sides state a year and they disagree, it is
not the same movie. An unknown year still never blocks.
Which is why the movie's OWN year is read with `releaseTagYear`, not
`extractYear`. The latter takes a year from anywhere in the title, which is the
right answer where a year is only a search hint that scoring will confirm — and
the wrong one here, where the year is part of an identity. "2001: A Space
Odyssey" is not a 2001 film: calling it one makes every genuine 1968 copy fail
the gate above, so the movie has no alternatives at all, and it moves the pin
key the moment enrichment supplies the real year. Only bracketed and trailing
forms count. The trailing form stays ambiguous on purpose ("Blade Runner 2049"
is a title, not a tag) — that is what the two-tier exact/base match is for.
## Why resolution is lazy
The `content` table stores no `container_extension`, and
@@ -513,23 +523,29 @@ table first and does nothing when the runtime rejects it, leaving the working
index in place — and does NOT record itself as done, so a later app version
shipping a newer SQLite upgrades then.
**Case folding for non-ASCII is still not possible.** `LOWER()`, GLOB classes
and the trigram tokenizer all fold ASCII only, so a Cyrillic title stored with
different capitalisation in two playlists ("ОН" vs "Он") cannot be matched by
any predicate available in stock SQLite. Closing that needs a stored
normalized-title column, which is deliberately out of scope here.
**Case folding for non-ASCII, on the FTS tier, is still not possible.**
`LOWER()` and the trigram tokenizer both fold ASCII only, so a Cyrillic title
stored with different capitalisation in two playlists ("ОН" vs "Он") cannot be
folded by the indexed path. Closing that needs a stored normalized-title
column, which is deliberately out of scope here.
The **scan** tier does fold it, because GLOB character classes are *not*
ASCII-only — `patternCompare` reads them as UTF-8 code points, and
`'Он' GLOB '*[Оо][Нн]*'` is true. `caseInsensitiveGlobPattern` therefore folds
the case in JavaScript, where Unicode case mapping is real, and hands SQLite one
`[lowerUpper]` class per character. It returns `null` — leaving the substring
tests as the whole answer — for a token holding a GLOB metacharacter (SQLite
GLOB has no escape character, so an unescaped `*` would become a wildcard) or a
case mapping that changes length (`ß` uppercases to `SS`).
SQLite's `LOWER()` and GLOB character classes are ASCII-only, so a short
non-ASCII title could not be folded or word-bounded and simply never matched —
the film stayed absent from the chip. The ASCII/Unicode branch is decided from the RAW token, not the normalized
one: normalization folds diacritics, so "Ça" arrives as "ca" and looks like
plain ASCII while the stored title still reads "Ça". ASCII tokens keep the
word-boundary GLOB
(what stops "it" matching "Titanic"); a non-ASCII token falls back to a
substring test against both the folded and the as-typed form. That covers a
title stored in the same case as the request or in lower case, and deliberately
does not claim full Unicode case folding. It is looser, and the normalized
The ASCII/Unicode branch is decided from the RAW token, not the normalized one:
normalization folds diacritics, so "Ça" arrives as "ca" and looks like plain
ASCII while the stored title still reads "Ça". ASCII tokens keep the
word-boundary GLOB (what stops "it" matching "Titanic"). The non-ASCII branch
keeps both substring tests alongside the folded class, because the class is
built from the raw token and cannot match across a diacritic difference, which
the folded-token test does. It has no word boundary — `[^a-z0-9]` would treat
every Cyrillic letter as a separator — so it is looser, and the normalized
confirmation afterwards is what makes looser safe.
## Claims about the present