mirror of
https://github.com/4gray/iptvnator.git
synced 2026-10-13 12:36:15 -08:00
fix(portals): match Greek titles whichever sigma the provider typed (#1310)
Greek Σ has two lowercase forms — medial σ and word-final ς — and neither the candidate query nor the confirmation treated them as one letter. The GLOB scan built each character's class from a one-way reach that only arrived at ς when it started from ς, so a request for "ΑΣ" never admitted a stored "Ας". Classes are now built from a fold group — every character sharing an uppercase form — derived by scanning the cased ranges at module load the way ACCENTED_BY_BASE already is. It generalises past sigma on its own: dotless ı folds with i, long ſ with s, historic Cyrillic letterforms with В Д О С Т Ъ Ѣ. Only the 24 groups of 767 that a per-character fold would miss are kept. Admitting the row was only half of it. normalizeTitleKeys then compared "ασ" against "ας" and discarded it, because toLowerCase picks the sigma form by position. Both SQL tiers already folded them together — SQLite's trigram tokenizer does full Unicode folding natively, unlike LOWER() — so the JS confirmation was the only tier that did not, making this a pre-existing gap on the FTS path as well. Normalization now rewrites ς to σ after lowercasing, which is what Unicode case folding does. Guards unchanged: a case mapping that changes length (ß → SS, İ) or a GLOB metacharacter still returns null rather than a partial pattern.
This commit is contained in:
1 parent
063662028a
commit
78df3e7dbb
8 files changed
+332
-95
No files matched your search
@@ -69,6 +69,25 @@ describe('normalizeTitleKeys', () => {
|
||||
it('keeps plural "seasons" endings — only singular markers are tags', () => {
|
||||
expect(normalizeTitle('Best of 2 Seasons')).toBe('best of 2 seasons');
|
||||
});
|
||||
|
||||
it('folds both lowercase spellings of Greek sigma together', () => {
|
||||
// `toLowerCase` picks the form by position — "ΑΣ" becomes "ας" while
|
||||
// an already-lowercase "ασ" stays medial — so one word arrives here
|
||||
// spelled two ways. Both SQL tiers fold them (SQLite's trigram
|
||||
// tokenizer natively, the scan's GLOB classes in JavaScript), so
|
||||
// leaving them apart here admits a candidate and then discards it.
|
||||
expect(normalizeTitle('ΑΣ')).toBe(normalizeTitle('Ας'));
|
||||
expect(normalizeTitle('ασ')).toBe(normalizeTitle('ας'));
|
||||
expect(normalizeTitle('ΑΣ')).toBe(normalizeTitle('ασ'));
|
||||
expect(normalizeTitle('Ο Άρχοντας')).toBe(normalizeTitle('ο αρχοντασ'));
|
||||
});
|
||||
|
||||
it('does not fold sigma into unrelated Greek letters', () => {
|
||||
// The fold is one letter's two lowercase forms, not a general
|
||||
// loosening of Greek — these must stay different titles.
|
||||
expect(normalizeTitle('ΑΣ')).not.toBe(normalizeTitle('ΑΝ'));
|
||||
expect(normalizeTitle('ΟΣ')).not.toBe(normalizeTitle('ΑΣ'));
|
||||
});
|
||||
});
|
||||
|
||||
describe('provider tag stripping', () => {
|
||||
|
||||
@@ -228,6 +228,15 @@ export function normalizeTitleKeys(
|
||||
.normalize('NFD')
|
||||
.replace(/[\u0300-\u036F]/g, '')
|
||||
.toLowerCase()
|
||||
// Greek \u03A3 has two lowercase forms and `toLowerCase` picks by position:
|
||||
// "\u0391\u03A3" becomes "\u03B1\u03C2" while an already-lowercase "\u03B1\u03C3" stays medial, so
|
||||
// the same word reaches this line spelled two ways. Both SQL tiers
|
||||
// fold them together \u2014 SQLite's trigram tokenizer does it natively,
|
||||
// and the scan's GLOB classes do it in `caseInsensitiveGlobPattern` \u2014
|
||||
// so without this the candidate is admitted by the query and then
|
||||
// thrown away by the confirmation. Folding to the medial form is what
|
||||
// Unicode case folding does.
|
||||
.replace(/\u03C2/g, '\u03C3')
|
||||
.replace(/[^\p{L}\p{N}]+/gu, ' ')
|
||||
.split(' ')
|
||||
.filter((token) => token !== '' && !QUALITY_TAGS.has(token))
|
||||
|
||||
Reference in new issue
Block a user