fix(portals): match Greek titles whichever sigma the provider typed (#1310)

Greek Σ has two lowercase forms — medial σ and word-final ς — and neither the
candidate query nor the confirmation treated them as one letter.

The GLOB scan built each character's class from a one-way reach that only
arrived at ς when it started from ς, so a request for "ΑΣ" never admitted a
stored "Ας". Classes are now built from a fold group — every character sharing
an uppercase form — derived by scanning the cased ranges at module load the way
ACCENTED_BY_BASE already is. It generalises past sigma on its own: dotless ı
folds with i, long ſ with s, historic Cyrillic letterforms with В Д О С Т Ъ Ѣ.
Only the 24 groups of 767 that a per-character fold would miss are kept.

Admitting the row was only half of it. normalizeTitleKeys then compared "ασ"
against "ας" and discarded it, because toLowerCase picks the sigma form by
position. Both SQL tiers already folded them together — SQLite's trigram
tokenizer does full Unicode folding natively, unlike LOWER() — so the JS
confirmation was the only tier that did not, making this a pre-existing gap on
the FTS path as well. Normalization now rewrites ς to σ after lowercasing,
which is what Unicode case folding does.

Guards unchanged: a case mapping that changes length (ß → SS, İ) or a GLOB
metacharacter still returns null rather than a partial pattern.
This commit is contained in:
4gray authored and GitHub committed 2026-07-29 23:14:59 +02:00
1 parent 063662028a
commit 78df3e7dbb
8 files changed
+332 -95

No files matched your search

@@ -69,6 +69,25 @@ describe('normalizeTitleKeys', () => {
it('keeps plural "seasons" endings — only singular markers are tags', () => {
expect(normalizeTitle('Best of 2 Seasons')).toBe('best of 2 seasons');
});
it('folds both lowercase spellings of Greek sigma together', () => {
// `toLowerCase` picks the form by position — "ΑΣ" becomes "ας" while
// an already-lowercase "ασ" stays medial — so one word arrives here
// spelled two ways. Both SQL tiers fold them (SQLite's trigram
// tokenizer natively, the scan's GLOB classes in JavaScript), so
// leaving them apart here admits a candidate and then discards it.
expect(normalizeTitle('ΑΣ')).toBe(normalizeTitle('Ας'));
expect(normalizeTitle('ασ')).toBe(normalizeTitle('ας'));
expect(normalizeTitle('ΑΣ')).toBe(normalizeTitle('ασ'));
expect(normalizeTitle('Ο Άρχοντας')).toBe(normalizeTitle('ο αρχοντασ'));
});
it('does not fold sigma into unrelated Greek letters', () => {
// The fold is one letter's two lowercase forms, not a general
// loosening of Greek — these must stay different titles.
expect(normalizeTitle('ΑΣ')).not.toBe(normalizeTitle('ΑΝ'));
expect(normalizeTitle('ΟΣ')).not.toBe(normalizeTitle('ΑΣ'));
});
});
describe('provider tag stripping', () => {
@@ -228,6 +228,15 @@ export function normalizeTitleKeys(
.normalize('NFD')
.replace(/[\u0300-\u036F]/g, '')
.toLowerCase()
// Greek \u03A3 has two lowercase forms and `toLowerCase` picks by position:
// "\u0391\u03A3" becomes "\u03B1\u03C2" while an already-lowercase "\u03B1\u03C3" stays medial, so
// the same word reaches this line spelled two ways. Both SQL tiers
// fold them together \u2014 SQLite's trigram tokenizer does it natively,
// and the scan's GLOB classes do it in `caseInsensitiveGlobPattern` \u2014
// so without this the candidate is admitted by the query and then
// thrown away by the confirmation. Folding to the medial form is what
// Unicode case folding does.
.replace(/\u03C2/g, '\u03C3')
.replace(/[^\p{L}\p{N}]+/gu, ' ')
.split(' ')
.filter((token) => token !== '' && !QUALITY_TAGS.has(token))