test(skills): validate repository skill contracts

This commit is contained in:
4gray committed 2026-07-30 11:36:43 +02:00
1 parent 829a51d571
commit 1e027b8e40
6 files changed
+641 -30

No files matched your search

+17 -29
View File
@@ -18,7 +18,13 @@ This file provides guidance to coding agents working in this repository.
- See `docs/architecture/nx-workspace-boundaries.md` for the current Nx tag and alias policy.
- ESLint enforces `max-lines` on TypeScript files: production code targets under 300 with a hard maximum of 400, while tests (`**/*.spec.ts`, `**/*.e2e.ts`, `apps/*-e2e/**`) are held to 1200 — a long spec signals coverage, not the design debt the production limit catches. Blank lines and comments are not counted, so a docblock never forces a split. Limits live in `tools/eslint/max-lines-config.mjs`, imported by both `eslint.config.mjs` and the generator so the rule and the baseline cannot drift. Files that predate the rule are baselined in `tools/eslint/max-lines-baseline.mjs`; after splitting a file, regenerate it with `node tools/eslint/generate-max-lines-baseline.mjs` (it runs ESLint's own rule rather than counting lines itself). Never add new files to the baseline — the list must only shrink. A new file that genuinely cannot be split (for example a function serialized into another process) instead carries its own file-wide `/* eslint-disable max-lines -- <why> */`; the generator skips those files, so a justified exemption never lands in the baseline. Remove such a directive once ESLint reports it as unused.
- Project `lint` targets that shell out to eslint must quote the glob, e.g. `eslint "apps/<project>/**/*.ts"`. An unquoted `**` is expanded by the POSIX shell on Linux and macOS (which has no `globstar`, so it matches only a shallow subset of files) while Windows passes the literal pattern to ESLint, which expands it recursively — the two hosts then lint different file sets. The target still reports success either way, so a broken glob hides missing coverage instead of failing. After changing such a target, compare the linted file count against `find <project> -name '*.ts' | wc -l`.
- Repository-specific skills are committed under `.codex/skills/`. Claude Code only discovers skills under `.claude/skills/`, so `release-notes` and `release-cut` are mirrored there and the two copies must be kept in sync; every other entry in `.claude/skills/` is personal and stays gitignored. If an external agent does not support skills, treat those files as concise ownership docs.
- Repository-specific skills live under `.codex/skills/`.
- Frontmatter descriptions are trigger-only and begin with `Use when`; keep
each skill at or below 500 words.
- Run `pnpm run skills:validate` after editing a committed skill or a literal
path it documents.
- Keep `.codex` and `.claude` copies of `release-notes` and `release-cut`
byte-identical.
## Documentation After Changes
@@ -410,35 +416,17 @@ Key files:
## Repo Skills
- `iptvnator-ui-design`
Repository-specific UI design guidance for IPTVnator.
Use when working on channel rows, EPG views, settings surfaces, shared selection styles, or light/dark theme consistency.
File: `.codex/skills/iptvnator-ui-design/SKILL.md`
- `.codex/skills/iptvnator-nx-architecture/SKILL.md`
- `.codex/skills/iptvnator-sqlite-db-worker/SKILL.md`
- `.codex/skills/iptvnator-theme-style/SKILL.md`
- `.codex/skills/iptvnator-ui-design/SKILL.md`
- `.codex/skills/release-cut/SKILL.md`
- `.codex/skills/release-notes/SKILL.md`
- `.codex/skills/stalker-portal/SKILL.md`
- `.codex/skills/xtream-electron/SKILL.md`
- `iptvnator-theme-style`
Theme architecture, design token reference, shared SCSS library, portal header pattern, Electron drag region, and common styling mistakes.
Use when adding/changing CSS tokens, styling portal headers or sidebars, using shared SCSS mixins (`portal-layout`, `content-grid`, `portal-sidebar`), or auditing cross-portal visual consistency.
File: `.codex/skills/iptvnator-theme-style/SKILL.md`
- `iptvnator-nx-architecture`
Repository-specific Nx monorepo structure, library placement rules, path alias guidance, and migration guardrails for portal/workspace/app code.
Use when deciding where code belongs, extracting code into libs, choosing imports, or refactoring Xtream/Stalker/Workspace boundaries.
File: `.codex/skills/iptvnator-nx-architecture/SKILL.md`
- `iptvnator-sqlite-db-worker`
Repository-specific guidance for the Electron non-EPG SQLite worker, including worker boundaries, request-scoped DB progress events, and validation steps for slow DB operations.
Use when moving heavy database work off the main thread, adding worker-backed SQLite operations, or wiring loading/progress UI for Xtream and playlist DB flows.
File: `.codex/skills/iptvnator-sqlite-db-worker/SKILL.md`
- `stalker-portal`
Repository-specific guidance for Stalker/Ministra catalogs, all three VOD/series modes, cross-surface `is_series` behavior, playback metadata, collections, EPG, and remote control.
Use when changing Stalker routes, stores, detail views, playback, favorites/recent activity, EPG, or remote control.
File: `.codex/skills/stalker-portal/SKILL.md`
- `xtream-electron`
Repository-specific guidance for IPTVnator's Electron-first Xtream implementation, including feature/data-access boundaries, worker-backed DB flows, and Xtream loading/progress UX expectations.
Use when working on Xtream routes, store/data-source logic, or Electron-backed Xtream import/search/delete behavior.
File: `.codex/skills/xtream-electron/SKILL.md`
Descriptions and trigger conditions are canonical in each skill's frontmatter;
do not duplicate them here.
<!-- nx configuration start-->
<!-- Leave the start & end comments to automatically receive updates. -->
+7 -1
View File
@@ -74,7 +74,13 @@ pnpm nx show projects
- Do not add new imports from legacy bare aliases such as `services`, `shared-interfaces`, `components`, `m3u-state`, or `database`.
- Every Nx project should keep `scope:*`, `domain:*`, and `type:*` tags in `project.json`.
- See `docs/architecture/nx-workspace-boundaries.md` for the current Nx tag and alias policy.
- Repository-specific skills are committed under `.codex/skills/`. Claude Code only discovers skills under `.claude/skills/`, so `release-notes` and `release-cut` are mirrored there and the two copies must be kept in sync; every other entry in `.claude/skills/` is personal and stays gitignored. If an agent does not load skills directly, treat those files as concise ownership docs.
- Repository-specific skills live under `.codex/skills/`.
- Frontmatter descriptions are trigger-only and begin with `Use when`; keep
each skill at or below 500 words.
- Run `pnpm run skills:validate` after editing a committed skill or a literal
path it documents.
- Keep `.codex` and `.claude` copies of `release-notes` and `release-cut`
byte-identical.
### Building and Serving
+1
View File
@@ -58,6 +58,7 @@
"serve:website": "nx serve website",
"build:website": "nx build website",
"i18n:check": "node tools/i18n/check-drift.mjs",
"skills:validate": "node tools/skills/validate-repository-skills.mjs",
"release:artwork:dry-run": "tsx --tsconfig tsconfig.base.json tools/release/generate-marketing-artwork.ts --dry-run",
"release:artwork:manifest": "tsx --tsconfig tsconfig.base.json tools/release/generate-marketing-artwork.ts --manifest",
"release:artwork:generate": "tsx --tsconfig tsconfig.base.json tools/release/generate-marketing-artwork.ts --generate",
+28
View File
@@ -0,0 +1,28 @@
{
"name": "repository-skills",
"$schema": "../../node_modules/nx/schemas/project-schema.json",
"projectType": "library",
"sourceRoot": "tools/skills",
"tags": ["scope:tools", "domain:skills", "type:tool"],
"targets": {
"test": {
"executor": "nx:run-commands",
"cache": true,
"inputs": [
"{workspaceRoot}/tools/skills/validate-repository-skills.mjs",
"{workspaceRoot}/tools/skills/validate-repository-skills.test.mjs"
],
"options": {
"command": "node --test tools/skills/validate-repository-skills.test.mjs",
"cwd": "{workspaceRoot}"
}
},
"lint": {
"executor": "nx:run-commands",
"options": {
"command": "node --check tools/skills/validate-repository-skills.mjs",
"cwd": "{workspaceRoot}"
}
}
}
}
+287
View File
@@ -0,0 +1,287 @@
import { access, readdir, readFile } from 'node:fs/promises';
import { isAbsolute, relative, resolve, sep } from 'node:path';
import { fileURLToPath } from 'node:url';
const SKILLS_ROOT = '.codex/skills';
const MIRRORED_SKILLS = ['release-cut', 'release-notes'];
const PATH_PREFIXES = [
'apps/',
'libs/',
'docs/',
'tools/',
'.changes/',
'.github/',
];
const ROOT_PATHS = new Set([
'package.json',
'pnpm-lock.yaml',
'nx.json',
'tsconfig.base.json',
'eslint.config.mjs',
'CHANGELOG.md',
'AGENTS.md',
'CLAUDE.md',
]);
const NON_LITERAL_PATH_CHARACTERS = /[*?[\]{}<>]/u;
function compareNames(left, right) {
if (left < right) return -1;
if (left > right) return 1;
return 0;
}
function unquoteScalar(value) {
if (value.length < 2) return value;
const firstCharacter = value[0];
const lastCharacter = value.at(-1);
const hasMatchingQuotes =
(firstCharacter === '"' || firstCharacter === "'") &&
lastCharacter === firstCharacter;
return hasMatchingQuotes ? value.slice(1, -1) : value;
}
function parseFrontmatter(markdown) {
const lines = markdown.split(/\r?\n/u);
if (lines[0] !== '---') {
return { fields: {}, continuedFields: new Set() };
}
const closingDelimiter = lines.indexOf('---', 1);
if (closingDelimiter === -1) {
return { fields: {}, continuedFields: new Set() };
}
const fields = {};
const continuedFields = new Set();
let activeField;
for (const line of lines.slice(1, closingDelimiter)) {
if (line.trim() === '') continue;
if (/^[ \t]/u.test(line)) {
if (activeField !== undefined) {
continuedFields.add(activeField);
}
continue;
}
const separator = line.indexOf(':');
if (separator <= 0) {
activeField = undefined;
continue;
}
const key = line.slice(0, separator).trim();
const value = line.slice(separator + 1).trim();
fields[key] = unquoteScalar(value);
activeField = key;
}
return { fields, continuedFields };
}
function countWords(markdown) {
const trimmedMarkdown = markdown.trim();
return trimmedMarkdown === ''
? 0
: trimmedMarkdown.split(/\s+/u).length;
}
function findLiteralRepositoryPaths(markdown) {
const paths = new Set();
for (const match of markdown.matchAll(/`([^`\r\n]+)`/gu)) {
const token = match[1];
const isRepositoryPath =
ROOT_PATHS.has(token) ||
PATH_PREFIXES.some((prefix) => token.startsWith(prefix));
if (isRepositoryPath && !NON_LITERAL_PATH_CHARACTERS.test(token)) {
paths.add(token);
}
}
return [...paths].sort(compareNames);
}
async function pathExists(path) {
try {
await access(path);
return true;
} catch (error) {
if (error?.code === 'ENOENT' || error?.code === 'ENOTDIR') {
return false;
}
throw error;
}
}
function isPathWithinRoot(rootDir, absolutePath) {
const relativePath = relative(resolve(rootDir), absolutePath);
return (
relativePath === '' ||
(!isAbsolute(relativePath) &&
relativePath !== '..' &&
!relativePath.startsWith(`..${sep}`))
);
}
async function readFileIfPresent(path) {
try {
return await readFile(path);
} catch (error) {
if (error?.code === 'ENOENT') return undefined;
throw error;
}
}
async function validateSkill({ rootDir, directoryName }) {
const relativePath = `${SKILLS_ROOT}/${directoryName}/SKILL.md`;
const markdown = await readFile(resolve(rootDir, relativePath), 'utf8');
const { fields: frontmatter, continuedFields } =
parseFrontmatter(markdown);
const diagnostics = [];
if (frontmatter.name !== directoryName) {
diagnostics.push(
`${relativePath}: frontmatter name "${frontmatter.name ?? ''}" must match directory "${directoryName}"`
);
}
if (
typeof frontmatter.description !== 'string' ||
frontmatter.description === '' ||
continuedFields.has('description')
) {
diagnostics.push(
`${relativePath}: description must be a one-line frontmatter value`
);
} else {
if (!frontmatter.description.startsWith('Use when')) {
diagnostics.push(
`${relativePath}: description must start with "Use when"`
);
}
if (frontmatter.description.length > 500) {
diagnostics.push(
`${relativePath}: description must be at most 500 characters (received ${frontmatter.description.length})`
);
}
}
const wordCount = countWords(markdown);
if (wordCount > 500) {
diagnostics.push(
`${relativePath}: skill must be at most 500 words (received ${wordCount})`
);
}
for (const referencedPath of findLiteralRepositoryPaths(markdown)) {
const absolutePath = resolve(rootDir, referencedPath);
if (!isPathWithinRoot(rootDir, absolutePath)) {
diagnostics.push(
`${relativePath}: referenced path escapes repository root: ${referencedPath}`
);
continue;
}
if (!(await pathExists(absolutePath))) {
diagnostics.push(
`${relativePath}: referenced path does not exist: ${referencedPath}`
);
}
}
return diagnostics;
}
async function validateReleaseMirrors(rootDir) {
const diagnostics = [];
for (const skillName of MIRRORED_SKILLS) {
const codexPath = `.codex/skills/${skillName}/SKILL.md`;
const claudePath = `.claude/skills/${skillName}/SKILL.md`;
const [codexContents, claudeContents] = await Promise.all([
readFileIfPresent(resolve(rootDir, codexPath)),
readFileIfPresent(resolve(rootDir, claudePath)),
]);
if (codexContents === undefined) {
diagnostics.push(
`${codexPath}: required release skill mirror is missing`
);
}
if (claudeContents === undefined) {
diagnostics.push(
`${claudePath}: required release skill mirror is missing`
);
}
if (
codexContents !== undefined &&
claudeContents !== undefined &&
!codexContents.equals(claudeContents)
) {
diagnostics.push(`${codexPath}: differs from ${claudePath}`);
}
}
return diagnostics;
}
export async function validateRepositorySkills({ rootDir }) {
const directoryEntries = (await readdir(resolve(rootDir, SKILLS_ROOT), {
withFileTypes: true,
}))
.filter((entry) => entry.isDirectory())
.sort((left, right) => compareNames(left.name, right.name));
const skillDirectories = [];
const diagnostics = [];
for (const entry of directoryEntries) {
if (
await pathExists(
resolve(rootDir, SKILLS_ROOT, entry.name, 'SKILL.md')
)
) {
skillDirectories.push(entry.name);
}
}
for (const directoryName of skillDirectories) {
diagnostics.push(
...(await validateSkill({ rootDir, directoryName }))
);
}
diagnostics.push(...(await validateReleaseMirrors(rootDir)));
return {
checkedSkills: skillDirectories.length,
diagnostics,
};
}
async function runCli() {
const { checkedSkills, diagnostics } = await validateRepositorySkills({
rootDir: process.cwd(),
});
if (diagnostics.length > 0) {
for (const diagnostic of diagnostics) {
console.error(diagnostic);
}
process.exitCode = 1;
return;
}
console.log(`Validated ${checkedSkills} repository skills.`);
}
const isCli =
process.argv[1] !== undefined &&
resolve(process.argv[1]) === fileURLToPath(import.meta.url);
if (isCli) {
await runCli();
}
@@ -0,0 +1,301 @@
import assert from 'node:assert/strict';
import {
mkdir,
mkdtemp,
rm,
symlink,
writeFile,
} from 'node:fs/promises';
import { tmpdir } from 'node:os';
import { basename, dirname, join } from 'node:path';
import test from 'node:test';
import { validateRepositorySkills } from './validate-repository-skills.mjs';
const validReleaseSkills = {
'release-cut': `---
name: release-cut
description: Use when preparing a repository release.
---
# Release Cut
`,
'release-notes': `---
name: release-notes
description: Use when deciding whether a change needs release notes.
---
# Release Notes
`,
};
async function writeFixtureFile(rootDir, relativePath, contents) {
const absolutePath = join(rootDir, relativePath);
await mkdir(dirname(absolutePath), { recursive: true });
await writeFile(absolutePath, contents);
}
async function createRepository(
t,
{
directoryName = 'example-skill',
name = directoryName,
description =
'"Use when deciding whether type: internal applies to a change."',
body = [
'# Example Skill',
'',
'Inspect `package.json` and `tools/existing.mjs`.',
'Examples such as `apps/*-e2e` and `libs/<domain>/feature` are not literal paths.',
'',
].join('\n'),
releaseMirrorTransform = (contents) => contents,
} = {}
) {
const rootDir = await mkdtemp(join(tmpdir(), 'repository-skills-'));
t.after(() => rm(rootDir, { recursive: true, force: true }));
await writeFixtureFile(rootDir, 'package.json', '{}\n');
await writeFixtureFile(rootDir, 'tools/existing.mjs', 'export {};\n');
await writeFixtureFile(
rootDir,
'.codex/skills/not-a-skill/README.md',
'No SKILL.md lives here.\n'
);
await writeFixtureFile(
rootDir,
`.codex/skills/${directoryName}/SKILL.md`,
`---
name: ${name}
description: ${description}
---
${body}`
);
await writeFixtureFile(
rootDir,
'.codex/skills/single-quoted-skill/SKILL.md',
`---
name: single-quoted-skill
description: 'Use when deciding whether type: internal applies to a release.'
---
# Single-Quoted Skill
`
);
for (const [skillName, contents] of Object.entries(validReleaseSkills)) {
await writeFixtureFile(
rootDir,
`.codex/skills/${skillName}/SKILL.md`,
contents
);
await writeFixtureFile(
rootDir,
`.claude/skills/${skillName}/SKILL.md`,
releaseMirrorTransform(contents, skillName)
);
}
return rootDir;
}
test('accepts single- and double-quoted colon-space descriptions and skips path examples', async (t) => {
const rootDir = await createRepository(t);
const result = await validateRepositorySkills({ rootDir });
assert.deepEqual(result, {
checkedSkills: 4,
diagnostics: [],
});
});
test('reports a frontmatter name that differs from its directory', async (t) => {
const rootDir = await createRepository(t, { name: 'different-name' });
const { diagnostics } = await validateRepositorySkills({ rootDir });
assert.deepEqual(diagnostics, [
'.codex/skills/example-skill/SKILL.md: frontmatter name "different-name" must match directory "example-skill"',
]);
});
test('reports a description that does not begin with Use when', async (t) => {
const rootDir = await createRepository(t, {
description: 'Repository-specific implementation guidance.',
});
const { diagnostics } = await validateRepositorySkills({ rootDir });
assert.deepEqual(diagnostics, [
'.codex/skills/example-skill/SKILL.md: description must start with "Use when"',
]);
});
test('reports a description longer than 500 characters', async (t) => {
const description = `Use when ${'x'.repeat(493)}`;
assert.equal(description.length, 502);
const rootDir = await createRepository(t, { description });
const { diagnostics } = await validateRepositorySkills({ rootDir });
assert.deepEqual(diagnostics, [
'.codex/skills/example-skill/SKILL.md: description must be at most 500 characters (received 502)',
]);
});
test('rejects an indented plain-scalar description continuation', async (t) => {
const rootDir = await createRepository(t, {
description: `Use when editing a skill.\n ${'x'.repeat(501)}`,
});
const { diagnostics } = await validateRepositorySkills({ rootDir });
assert.deepEqual(diagnostics, [
'.codex/skills/example-skill/SKILL.md: description must be a one-line frontmatter value',
]);
});
test('rejects a blank-separated plain-scalar description continuation', async (t) => {
const rootDir = await createRepository(t, {
description: `Use when editing a skill.\n\n ${'x'.repeat(501)}`,
});
const { diagnostics } = await validateRepositorySkills({ rootDir });
assert.deepEqual(diagnostics, [
'.codex/skills/example-skill/SKILL.md: description must be a one-line frontmatter value',
]);
});
test('allows a blank frontmatter separator without treating it as a continuation', async (t) => {
const rootDir = await createRepository(t, {
description: 'Use when editing a skill.\n ',
});
const { diagnostics } = await validateRepositorySkills({ rootDir });
assert.deepEqual(diagnostics, []);
});
test('reports a complete skill longer than 500 words', async (t) => {
const rootDir = await createRepository(t, {
body: `# Example Skill\n\n${Array.from({ length: 501 }, () => 'word').join(' ')}`,
});
const { diagnostics } = await validateRepositorySkills({ rootDir });
assert.equal(diagnostics.length, 1);
assert.match(
diagnostics[0],
/^\.codex\/skills\/example-skill\/SKILL\.md: skill must be at most 500 words \(received \d+\)$/
);
});
test('reports a backticked literal repository path that does not exist', async (t) => {
const rootDir = await createRepository(t, {
body: '# Example Skill\n\nInspect `docs/missing-file.md`.',
});
const { diagnostics } = await validateRepositorySkills({ rootDir });
assert.deepEqual(diagnostics, [
'.codex/skills/example-skill/SKILL.md: referenced path does not exist: docs/missing-file.md',
]);
});
test('rejects a literal path that escapes the repository even when it exists', async (t) => {
const rootDir = await createRepository(t);
const outsideName = `${basename(rootDir)}-outside.md`;
const outsidePath = join(dirname(rootDir), outsideName);
const referencedPath = `docs/../../${outsideName}`;
t.after(() => rm(outsidePath, { force: true }));
await writeFile(outsidePath, 'outside\n');
await writeFixtureFile(
rootDir,
'.codex/skills/example-skill/SKILL.md',
`---
name: example-skill
description: Use when validating a literal path.
---
Inspect \`${referencedPath}\`.
`
);
const { diagnostics } = await validateRepositorySkills({ rootDir });
assert.deepEqual(diagnostics, [
`.codex/skills/example-skill/SKILL.md: referenced path escapes repository root: ${referencedPath}`,
]);
});
test(
'propagates non-missing path access failures',
{ skip: process.platform === 'win32' },
async (t) => {
const rootDir = await createRepository(t, {
body: '# Example Skill\n\nInspect `docs/loop/file.md`.',
});
await mkdir(join(rootDir, 'docs'));
await symlink('loop', join(rootDir, 'docs/loop'));
await assert.rejects(
validateRepositorySkills({ rootDir }),
(error) => {
assert.equal(error.code, 'ELOOP');
return true;
}
);
}
);
test('reports byte-different release skill mirrors', async (t) => {
const rootDir = await createRepository(t, {
releaseMirrorTransform: (contents, skillName) =>
skillName === 'release-notes' ? `${contents}\n` : contents,
});
const { diagnostics } = await validateRepositorySkills({ rootDir });
assert.deepEqual(diagnostics, [
'.codex/skills/release-notes/SKILL.md: differs from .claude/skills/release-notes/SKILL.md',
]);
});
test('reports a missing release skill mirror without stopping validation', async (t) => {
const rootDir = await createRepository(t, {
description: 'Implementation guidance.',
});
await rm(
join(rootDir, '.claude/skills/release-cut/SKILL.md')
);
const { diagnostics } = await validateRepositorySkills({ rootDir });
assert.deepEqual(diagnostics, [
'.codex/skills/example-skill/SKILL.md: description must start with "Use when"',
'.claude/skills/release-cut/SKILL.md: required release skill mirror is missing',
]);
});
test('returns every diagnostic in deterministic order', async (t) => {
const rootDir = await createRepository(t, {
name: 'wrong-name',
description: 'Implementation guidance.',
body: 'Inspect `docs/z-missing.md` and `docs/a-missing.md`.',
releaseMirrorTransform: (contents, skillName) =>
skillName === 'release-notes' ? `${contents}\n` : contents,
});
const { diagnostics } = await validateRepositorySkills({ rootDir });
assert.deepEqual(diagnostics, [
'.codex/skills/example-skill/SKILL.md: frontmatter name "wrong-name" must match directory "example-skill"',
'.codex/skills/example-skill/SKILL.md: description must start with "Use when"',
'.codex/skills/example-skill/SKILL.md: referenced path does not exist: docs/a-missing.md',
'.codex/skills/example-skill/SKILL.md: referenced path does not exist: docs/z-missing.md',
'.codex/skills/release-notes/SKILL.md: differs from .claude/skills/release-notes/SKILL.md',
]);
});