build(site): [SITE-07] reproduce all console modules and Parcel loader

scripts/site-vendor/README.md

@@ -57,8 +57,9 @@ output through PowerShell text redirection. No dependency installation is needed
 
 The desktop console's owned TS entry/view and lifecycle now build through
 `build:console` / `check:console`; see [console maintenance](console/README.md).
-Its other 100 Parcel modules remain frozen pending full provenance and notices.
+Its other 100 Parcel modules remain frozen and now reproduce exactly from fixed
+archives, including the Parcel loader. Full embedded attribution/notices remain open.
 
 Follow-up: finish Monaco's broader bundled-component notice audit, the console
-vendor provenance and remaining fonts/media. Do not upgrade these assets
+embedded attribution/notices and remaining fonts/media. Do not upgrade these assets
 without verifying globals, AMD/worker paths, CSS, consoleLog and user interaction.

scripts/site-vendor/console/README.md

@@ -64,8 +64,9 @@ Historical browser cases intentionally reproduce frozen defects; candidate cases
 prove their repairs independently. Unit tests check shared ownership, failed
 installation, callback errors, object identity, exact CSS and vendor preservation.
 No new dependency is required. Existing esbuild 0.27.7 and TypeScript 5.9.3 provide
-generation and structural parsing. Full console dependency provenance and notices
-remain SITE-07 / VENDOR-08; this build does not close them. See
+generation and structural parsing. All vendor module bodies now have exact
+reconstruction evidence; full embedded attribution and notices remain
+SITE-07 / VENDOR-08. See
 `refactor/console-modernization.md` for contracts and evidence.
 
 ## Historical source reconstruction
@@ -76,8 +77,9 @@ Their rebuilt bodies exactly match the frozen bundle. The separate
 `console-commonjs-provenance.json` adds 41 exact modules from 13 official archives:
 React/ReactDOM, scheduler, object-assign, react-is, prop-types, shallowequal,
 process, hoist-non-react-statics, is-dom/is-object/is-window and linkifyjs.
-The 27 remaining vendor modules and Parcel wrapper need further provenance; this does not establish a
-unique original installed version or recover its missing lockfile.
+`console-esm-provenance.json` adds the final 27 modules from 18 archives and the
+Parcel 1.12.5 prelude/invocation. All 100 vendor bodies match; this does not
+establish unique original installed versions or recover the missing lockfile.
 
 ```sh
 node scripts/site-vendor/console/reproduce.ts --fetch
@@ -85,24 +87,35 @@ node scripts/site-vendor/console/reproduce.ts
 yarn test:site-console
 ```
 
-The first command downloads 15 pinned npm archives into the dedicated ignored
+The first command downloads 35 pinned npm archives into the dedicated ignored
 `refactor/.cache/console-feed-reproduction/` directory. The second uses that cache
 offline. Both verify SHA-512 SRI, archive SHA-256, source member fingerprints,
-compiler bytes, original notice bytes and exact output of all 73 identified
-modules. They load Terser 3.17.0 as a historical build
-tool with the already installed source-map 0.6.1; they do not install dependencies,
-change Yarn or execute the archived console-feed runtime. Canonical Node is required.
+compiler bytes, original notice bytes and exact output of all 100 identified
+modules. They load Terser 3.17.0 with the installed source-map 0.6.1, and the
+self-contained Babel 7.16.4 archive. They do not install dependencies, change Yarn
+or execute the archived runtime libraries. Canonical Node is required.
 The minification recipe is recovered from Parcel 1.12.5's archive; this proves a
 reproducing transform, not that the original author used precisely that Parcel
 version. `provenance.ts` separately verifies complete traversal, relative source
 mapping and external dependency boundaries. Missing, unreachable, duplicate or
 changed evidence fails instead of silently shrinking the comparison. Multiple
 package roots are explicit, relative edges must stay in the same archive, and
-the remaining module IDs are checked as a complete list. Production entrypoints
+all module IDs are covered once. Named/scoped imports must resolve to the
+identified package. Production entrypoints
 use the pinned-source process.env.NODE_ENV substitution. The process shim's
 single process.browser assignment is removed following Parcel's original visitor;
 this is checked against exact source/output bytes, not a general JS rewrite.
 
+`reconstruction.ts` owns the ordered Babel stages, explicit historical environment
+and Parcel global injection. Most ESM inputs need only CommonJS conversion.
+Styled-components needs a separate earlier typeof-symbol pass; combining the
+passes also changes newly generated interop helpers and fails exact comparison.
+Parcel adds globals before minification and again before final output. Preserve
+that sequence, including the observed process/define declarations. Compiler
+options are cloned per module because historical Terser mutates them.
+`reproduce.ts` orchestrates archives, hashes and all three source groups; normal
+site builds continue using the frozen verified vendor boundary.
+
 The archived LICENSE is preserved verbatim with its Facebook attribution under
 `refactor/baselines/site-vendor/console-feed-3.2.2-LICENSE.txt`. Do not rewrite it
 or infer it covers every embedded/external component. Complete notices remain
@@ -110,5 +123,12 @@ open, including the bundled replicator and remaining dependencies.
 The other 13 archives' LICENSE texts are also frozen under
 `refactor/baselines/site-vendor/console-commonjs/`, with exact bytes preserved by
 Git attributes. They are source evidence; complete site notice delivery still
-requires the remaining component review. React-inspector 5.1.1's CJS file did
-not match; its ESM conversion remains an investigation, not an accepted source.
+requires the remaining component review. The final ESM and Parcel texts live in
+`refactor/baselines/site-vendor/console-esm/`. React-inspector 5.1.1's ESM member
+is an exact match after historical conversion; its CJS member was a failed
+candidate. Styled-components 5.3.3 omits LICENSE in npm; the supplemental original
+comes from fixed upstream commit 9b3457036cfedf1d5336f654f3171657630a9fd8.
+The fetch command verifies that immutable upstream text through GitHub's Contents
+API too (the raw URL had connection resets); both URLs and the blob ID are
+recorded, and decoded content must match the same hash. Offline mode checks
+the frozen bytes. Full embedded-component attribution remains open.

scripts/site-vendor/console/provenance.ts

@@ -84,3 +84,16 @@ export function verifyModules<S extends Source>(modules: Map<string, Module>, so
   assert.deepEqual(dependencies, external, 'External dependency boundary changed')
   return visited.size
 }
+
+export function verifyPackageEdges(modules: Map<string, Module>, owners: { id: string, packageName: string }[]) {
+  const packages = new Map(owners.map(owner => [owner.id, owner.packageName]))
+  assert.equal(packages.size, owners.length, 'Duplicate package ownership')
+  for (const owner of owners) {
+    const module = modules.get(owner.id)
+    assert(module, `Unknown owned module: ${owner.id}`)
+    for (const [request, child] of Object.entries(module.dependencies)) {
+      const expected = request.startsWith('.') ? owner.packageName : request.split('/').slice(0, request.startsWith('@') ? 2 : 1).join('/')
+      assert.equal(packages.get(child), expected, `Wrong package for ${owner.id} -> ${request}`)
+    }
+  }
+}

scripts/site-vendor/console/reconstruction.ts

@@ -0,0 +1,33 @@
+import type { Source } from './provenance.ts'
+import assert from 'node:assert/strict'
+
+export interface EsmSource extends Source {
+  archive: string
+  stages: string[][]
+  environment: Record<string, string | null>
+  prefix: string
+}
+export interface BabelRuntime {
+  version: string
+  transform: (source: string, options: { plugins: string[] }) => { code: string }
+}
+
+export function reconstructModule(source: string, item: EsmSource, babel: BabelRuntime, minify: (source: string) => string) {
+  for (const plugins of item.stages) {
+    assert(plugins.length && plugins.every(name => ['transform-typeof-symbol', 'transform-modules-commonjs'].includes(name)), 'Unexpected historical transform')
+    source = babel.transform(source, { plugins }).code
+  }
+  for (const [name, value] of Object.entries(item.environment)) {
+    assert(['NODE_ENV', 'REACT_APP_SC_ATTR', 'SC_ATTR', 'REACT_APP_SC_DISABLE_SPEEDY', 'SC_DISABLE_SPEEDY'].includes(name), 'Unexpected historical environment key')
+    source = source.replaceAll(`process.env.${name}`, value === null ? 'undefined' : JSON.stringify(value))
+  }
+  assert(['', 'var define;\n', 'var process = require("process");\n'].includes(item.prefix), 'Unexpected Parcel global injection')
+  // Parcel generate() adds globals for minification and again for final output.
+  return item.prefix + minify(item.prefix + source)
+}
+
+export function verifyPrelude(bundle: string, prelude: string, footer: string) {
+  const prefix = prelude.trim().replace(/;$/, '')
+  assert(bundle.startsWith(`${prefix}({`), 'Parcel prelude differs')
+  assert(bundle.endsWith(footer), 'Parcel invocation or map trailer differs')
+}

scripts/site-vendor/console/reproduce.ts

@@ -1,12 +1,15 @@
 import type { Archive, External, Source } from './provenance.ts'
+import type { BabelRuntime, EsmSource } from './reconstruction.ts'
 import assert from 'node:assert/strict'
+import { Buffer } from 'node:buffer'
 import { execFileSync } from 'node:child_process'
 import fs from 'node:fs'
 import { createRequire } from 'node:module'
 import path from 'node:path'
 import process from 'node:process'
 import { fileURLToPath } from 'node:url'
-import { hash, parcelModules, verifyArchive, verifyModules } from './provenance.ts'
+import { hash, parcelModules, verifyArchive, verifyModules, verifyPackageEdges } from './provenance.ts'
+import { reconstructModule, verifyPrelude } from './reconstruction.ts'
 
 interface CompilerOptions { warnings: boolean, safari10: boolean, mangle: { toplevel: boolean }, output?: { comments: boolean } }
 interface Provenance {
@@ -18,30 +21,47 @@ interface Provenance {
 }
 interface Minifier { minify: (source: string, options: CompilerOptions) => { code?: string, error?: unknown } }
 interface CommonSource extends Source { archive: string, transform: 'plain' | 'production' | 'process-browser' }
-interface CommonProvenance {
-  archives: (Archive & { id: string, notices: { source: string, member: string, sha256: string }[] })[]
-  sources: CommonSource[]
+interface Notice { source: string, member: string, sha256: string }
+interface SourceGroup<S extends Source> {
+  archives: (Archive & { id: string, notices: Notice[] })[]
+  sources: S[]
   external: External[]
   roots: string[]
   compilerOptions: CompilerOptions
   unresolvedModules: string[]
 }
+interface EsmProvenance extends SourceGroup<EsmSource> {
+  babel: { archive: Archive, member: string, sha256: string, version: string }
+  parcel: { archive: Archive, prelude: { member: string, sha256: string }, footer: string, notices: Notice[], recipeMembers: { member: string, sha256: string }[] }
+  supplementalNotices: { url: string, apiUrl?: string, source: string, sha256: string }[]
+}
 
 const root = fileURLToPath(new URL('../../../', import.meta.url))
 const record: Provenance = JSON.parse(fs.readFileSync(path.join(root, 'refactor/baselines/console-feed-provenance.json'), 'utf8'))
-const common: CommonProvenance = JSON.parse(fs.readFileSync(path.join(root, 'refactor/baselines/console-commonjs-provenance.json'), 'utf8'))
+const common: SourceGroup<CommonSource> = JSON.parse(fs.readFileSync(path.join(root, 'refactor/baselines/console-commonjs-provenance.json'), 'utf8'))
+const esm: EsmProvenance = JSON.parse(fs.readFileSync(path.join(root, 'refactor/baselines/console-esm-provenance.json'), 'utf8'))
 assert(process.argv.slice(2).every(arg => arg === '--fetch'), 'Use reproduce.ts [--fetch]')
 assert.equal(process.version, `v${fs.readFileSync(path.join(root, '.node-version'), 'utf8').trim()}`, 'Use canonical Node')
 const cacheRoot = fs.realpathSync(path.join(root, 'refactor/.cache'))
 const cache = path.join(cacheRoot, 'console-feed-reproduction')
 fs.mkdirSync(cache, { recursive: true })
 assert.equal(fs.realpathSync(cache), cache, 'Reproduction cache must not redirect')
-for (const archive of [record.archive, record.compiler.archive, ...common.archives]) {
-  const target = path.join(cache, `${archive.name}-${archive.version}.tgz`)
+async function download(url: string) {
+  try {
+    const response = await fetch(url)
+    assert(response.ok, `HTTP ${response.status}`)
+    return new Uint8Array(await response.arrayBuffer())
+  }
+  catch (cause) {
+    const error = new Error(`Historical source download failed: ${url}`)
+    Object.defineProperty(error, 'cause', { value: cause })
+    throw error
+  }
+}
+for (const archive of [record.archive, record.compiler.archive, ...common.archives, ...esm.archives, esm.babel.archive, esm.parcel.archive]) {
+  const target = path.join(cache, `${encodeURIComponent(archive.name)}-${archive.version}.tgz`)
   if (process.argv.includes('--fetch')) {
-    const response = await fetch(archive.tarball)
-    assert(response.ok, `Archive request failed: ${response.status}`)
-    const bytes = new Uint8Array(await response.arrayBuffer())
+    const bytes = await download(archive.tarball)
     verifyArchive(bytes, archive)
     fs.writeFileSync(target, bytes)
   }
@@ -57,11 +77,18 @@ assert.equal(require('source-map/package.json').version, record.compiler.sourceM
 
 const { minify } = require(compilerPath) as Minifier
 
+const babelBytes = readMember(`${encodeURIComponent(esm.babel.archive.name)}-${esm.babel.archive.version}.tgz`, esm.babel.member)
+assert.equal(hash(babelBytes), esm.babel.sha256, 'Babel compiler member changed')
+const babelPath = path.join(cache, 'babel.cjs')
+fs.writeFileSync(babelPath, babelBytes)
+const babel = require(babelPath) as BabelRuntime
+
+assert.equal(babel.version, esm.babel.version, 'Babel version changed')
 const frozen = fs.readFileSync(path.join(root, record.frozen.path), 'utf8')
 assert.equal(hash(frozen), record.frozen.sha256, 'Frozen console changed')
 assert.equal(record.sources.length, 32, 'Incomplete console-feed scope')
 function compile(source: string, options: CompilerOptions) {
-  const result = minify(source, options)
+  const result = minify(source, structuredClone(options))
   if (result.error)
     throw result.error
   assert(typeof result.code === 'string')
@@ -70,10 +97,11 @@ function compile(source: string, options: CompilerOptions) {
 const modules = parcelModules(frozen)
 const count = verifyModules(modules, record.sources, record.external, member => readMember('console-feed-3.2.2.tgz', member).toString('utf8'), source => compile(source, record.compiler.options))
 assert.equal(common.sources.length, 41, 'Incomplete commonjs scope')
-const archiveIds = new Set(common.archives.map(archive => archive.id))
-assert.equal(archiveIds.size, common.archives.length, 'Duplicate archives')
-for (const archive of common.archives) {
-  assert.equal(archive.id, `${archive.name}-${archive.version}`, 'Archive ID differs')
+const runtimeArchives = [...common.archives, ...esm.archives]
+const archiveIds = new Set(runtimeArchives.map(archive => archive.id))
+assert.equal(archiveIds.size, runtimeArchives.length, 'Duplicate archives')
+for (const archive of runtimeArchives) {
+  assert.equal(archive.id, `${encodeURIComponent(archive.name)}-${archive.version}`, 'Archive ID differs')
   for (const notice of archive.notices) {
     const bytes = readMember(`${archive.id}.tgz`, notice.member)
     assert.equal(hash(bytes), notice.sha256, 'Upstream notice differs')
@@ -100,4 +128,44 @@ const commonCount = verifyModules(modules, common.sources, common.external, (mem
 const identified = new Set([...record.sources, ...common.sources].map(source => source.id))
 assert.equal(identified.size, count + commonCount, 'Duplicate identified module')
 assert.deepEqual([...modules.keys()].filter(id => !identified.has(id) && !['Focm', 'W5CS'].includes(id)), common.unresolvedModules, 'Unresolved module scope changed')
-console.log(JSON.stringify({ exactModules: identified.size, consoleFeed: count, commonjs: commonCount, totalVendorModules: 100, unresolved: common.unresolvedModules.length, licenseClosure: false }))
+assert.equal(esm.sources.length, 27, 'Incomplete ESM source scope')
+const esmCount = verifyModules(modules, esm.sources, esm.external, (member, item) => {
+  assert(archiveIds.has(item.archive), 'Unknown ESM source archive')
+  return readMember(`${item.archive}.tgz`, member).toString('utf8')
+}, (source, item) => reconstructModule(source, item, babel, code => compile(code, esm.compilerOptions)), { roots: esm.roots, sourcePrefix: 'package/' })
+for (const source of esm.sources) {
+  assert(!identified.has(source.id), 'Duplicate ESM module')
+  identified.add(source.id)
+}
+assert.equal(identified.size, 100, 'Incomplete vendor reconstruction')
+assert.deepEqual([...modules.keys()].filter(id => !identified.has(id) && !['Focm', 'W5CS'].includes(id)), esm.unresolvedModules, 'Unresolved ESM scope changed')
+assert.equal(esm.unresolvedModules.length, 0, 'Vendor source scope still incomplete')
+const archiveNames = new Map(runtimeArchives.map(archive => [archive.id, archive.name]))
+verifyPackageEdges(modules, [
+  ...record.sources.map(source => ({ id: source.id, packageName: record.archive.name })),
+  ...[...common.sources, ...esm.sources].map(source => ({ id: source.id, packageName: archiveNames.get(source.archive)! })),
+])
+const parcelArchive = `${esm.parcel.archive.name}-${esm.parcel.archive.version}.tgz`
+const prelude = readMember(parcelArchive, esm.parcel.prelude.member)
+assert.equal(hash(prelude), esm.parcel.prelude.sha256, 'Parcel prelude source changed')
+verifyPrelude(frozen, prelude.toString('utf8'), esm.parcel.footer)
+for (const recipe of esm.parcel.recipeMembers)
+  assert.equal(hash(readMember(parcelArchive, recipe.member)), recipe.sha256, 'Parcel recipe source changed')
+for (const notice of esm.parcel.notices) {
+  assert.equal(hash(readMember(parcelArchive, notice.member)), notice.sha256, 'Parcel notice source changed')
+  assert.equal(hash(fs.readFileSync(path.join(root, notice.source))), notice.sha256, 'Frozen Parcel notice changed')
+}
+for (const notice of esm.supplementalNotices) {
+  assert.equal(hash(fs.readFileSync(path.join(root, notice.source))), notice.sha256, 'Frozen supplemental notice changed')
+  if (process.argv.includes('--fetch')) {
+    let bytes: Uint8Array = await download(notice.apiUrl || notice.url)
+    if (notice.apiUrl) {
+      const content: { encoding: string, content: string } = JSON.parse(Buffer.from(bytes).toString('utf8'))
+      assert.equal(content.encoding, 'base64', 'Unexpected upstream notice encoding')
+      assert(typeof content.content === 'string', 'Missing upstream notice content')
+      bytes = Buffer.from(content.content, 'base64')
+    }
+    assert.equal(hash(bytes), notice.sha256, 'Upstream supplemental notice changed')
+  }
+}
+console.log(JSON.stringify({ exactModules: identified.size, consoleFeed: count, commonjs: commonCount, esm: esmCount, parcelPrelude: true, packageEdges: true, unresolved: 0, licenseClosure: false }))