{"$schema":"https://doc-kit.nodejs.org/schemas/api-doc/1.0.1.json","id":"contributing-comparators","path":"/contributing/comparators","type":"module","module":"contributing-comparators","title":"Adding a comparator","introducedIn":null,"sourceLink":null,"stability":null,"added":[],"deprecated":[],"removed":[],"napiVersion":[],"changes":[],"description":"A comparator is a small Node.js script that compares one generator's output on\n`main` with its output on a pull request and prints a Markdown report. CI\ncollects these reports into a comment on the pull request, so you and the\nreviewers can see what a change does to the generated docs, and to generation\ntime and memory, before it lands.\n\nComparators live in `scripts/comparators/` and run only in this repository's\nCI. They are not part of any published package.","summary":"A comparator is a small Node.js script that compares one generator's output on `main` with its output on a pull request and prints a Markdown report. CI collects these reports into a comment on the pull request, so you and the reviewers can see what a change does to the generated docs, and to generation time and memory, before it lands.","examples":[],"children":[{"kind":"section","id":"how-the-comparison-runs","name":"How the comparison runs","title":"How the comparison runs","scope":"module","overloadOf":null,"stability":null,"added":[],"deprecated":[],"removed":[],"napiVersion":[],"changes":[],"description":"Two workflows work together:\n\n1. `.github/workflows/generate.yml` (\"Generate and Compare Docs\") runs on every\n   push to `main`. It records the latest nodejs/node commit, generates every\n   target in its matrix from that commit's docs, and uploads each target's\n   output directory as an artifact named after the target.\n2. On a pull request, the same workflow finds the `main` run for the pull\n   request's base commit and reuses its Node.js commit. Both sides are built\n   from the same sources, so any difference comes from the pull request.\n3. Each matrix entry generates into `out/`. Those with a `compare` field then\n   download the target's artifact from the `main` run into `base/` and run\n   `node scripts/comparators/<compare>.mjs > out/comparison.txt`.\n4. When the workflow finishes, `.github/workflows/leave-comment.yml` joins\n   every `comparison.txt` and posts the result as a single comment on the pull\n   request (tagged `compared`, so later pushes update it). If every report is\n   empty, it posts nothing.\n\nEach generation runs under `/usr/bin/time`, which writes `out/benchmark.json`\nwith the elapsed time, user and system CPU time, and peak resident memory.\nComparators read it for their performance section and leave it out of the\noutput comparison.\n\nMatrix entries without `compare` (at the moment `man-page`, `addon-verify` and\n`json-simple`) are generated and uploaded, but not compared.","summary":"Two workflows work together:","examples":[],"children":[]},{"kind":"section","id":"built-in-comparators","name":"Built-in comparators","title":"Built-in comparators","scope":"module","overloadOf":null,"stability":null,"added":[],"deprecated":[],"removed":[],"napiVersion":[],"changes":[],"description":"| Comparator         | What it reports                                                                                                    | Used by                                                                                                                                                           |\n| ------------------ | ------------------------------------------------------------------------------------------------------------------ | ----------------------------------------------------------------------------------------------------------------------------------------------------------------- |\n| `file-size`        | Files added, removed or changed in size, largest change first, with the net change, in a collapsible table         | `web` (the former name of [`html`](../generators/html.html)), [`orama-db`](../generators/orama-db.html), [`llms-txt`](../generators/llms-txt.html), `legacy-html` |\n| `object-assertion` | Files added or removed, and an `assert.deepStrictEqual` diff for each file whose parsed JSON differs, one per file | [`json`](../generators/json.html), [`api-links`](../generators/api-links.html), `legacy-json`                                                                     |\n\nBoth end with a **Performance estimate** section: the generation time and peak\nmemory of each side, read from `benchmark.json`. It comes from a single CI run,\nso treat it as a rough signal. It's left out when either side has no\n`benchmark.json`.\n\nKeep their limits in mind when you pick one:\n\n* `file-size` compares sizes only. An edit that keeps a file's size the same\n  doesn't show up.\n* `object-assertion` parses every output file as JSON, so it only suits\n  generators whose output is all JSON.","summary":"Both end with a **Performance estimate** section: the generation time and peak memory of each side, read from `benchmark.json`. It comes from a single CI run, so treat it as a rough signal. It's left out when either side has no `benchmark.json`.","examples":[],"children":[]},{"kind":"section","id":"shared-helpers","name":"Shared helpers","title":"Shared helpers","scope":"module","overloadOf":null,"stability":null,"added":[],"deprecated":[],"removed":[],"napiVersion":[],"changes":[],"description":"| Import                              | From                                  | What it gives you                                                                                                                                                              |\n| ----------------------------------- | ------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |\n| `BASE`, `HEAD`                      | `scripts/constants.mjs`               | The two build directories: `base/` and `out/` at the repository root (both gitignored), unless the `BASE` and `HEAD` environment variables point elsewhere.                    |\n| `TITLE`                             | `scripts/constants.mjs`               | The report heading, ``## `<GENERATOR>` Generator``, built from the `GENERATOR` environment variable, which CI sets to the target name. Set the `TITLE` variable to replace it. |\n| `BENCHMARK_FILE`, `COMPARISON_FILE` | `scripts/constants.mjs`               | The metadata file names, `benchmark.json` and `comparison.txt`.                                                                                                                |\n| `listOutputFiles(directory)`        | `scripts/comparators/files.mjs`       | Every file under `directory`, recursively, as sorted relative paths, without the two metadata files.                                                                           |\n| `comparePerformance()`              | `scripts/comparators/performance.mjs` | The performance section as Markdown, or an empty string when either side has no `benchmark.json`. It throws if a benchmark value isn't a finite number.                        |","summary":"","examples":[],"children":[]},{"kind":"section","id":"writing-a-comparator","name":"Writing a comparator","title":"Writing a comparator","scope":"module","overloadOf":null,"stability":null,"added":[],"deprecated":[],"removed":[],"napiVersion":[],"changes":[],"description":"A comparator is an ES module in `scripts/comparators/`. Its file name without\n`.mjs` is the value of the `compare` field in the workflow matrix. It should:\n\n* Read the two builds from `BASE` and `HEAD`, listing their files with\n  `listOutputFiles`, which skips `benchmark.json` and the `comparison.txt`\n  that the report is being written to.\n* Print Markdown to standard output, starting with `TITLE`, and print nothing\n  when there's nothing to report.\n* Report differences instead of failing on them. A comparator that exits with\n  an error fails the CI job, so throw only when something is actually broken.\n* End with `comparePerformance()`, like the built-in comparators, so the\n  timing stays visible.","summary":"A comparator is an ES module in `scripts/comparators/`. Its file name without `.mjs` is the value of the `compare` field in the workflow matrix. It should:","examples":[],"children":[{"kind":"section","id":"create-the-script","name":"Create the script","title":"Create the script","scope":"module","overloadOf":null,"stability":null,"added":[],"deprecated":[],"removed":[],"napiVersion":[],"changes":[],"description":"This example lists every file whose bytes changed, which catches the\nsame-size edits that `file-size` misses:\n\n```mjs displayName=\"scripts/comparators/changed-files.mjs\"\nimport { readFile } from 'node:fs/promises';\nimport { join } from 'node:path';\n\nimport { BASE, HEAD, TITLE } from '../constants.mjs';\nimport { listOutputFiles } from './files.mjs';\nimport { comparePerformance } from './performance.mjs';\n\nconst [baseFiles, headFiles] = await Promise.all(\n  [BASE, HEAD].map(directory => listOutputFiles(directory))\n);\nconst baseSet = new Set(baseFiles);\nconst headSet = new Set(headFiles);\n\nconst describe = async file => {\n  if (!baseSet.has(file)) {\n    return `- \\`${file}\\` added`;\n  }\n\n  if (!headSet.has(file)) {\n    return `- \\`${file}\\` removed`;\n  }\n\n  const [base, head] = await Promise.all(\n    [BASE, HEAD].map(directory => readFile(join(directory, file)))\n  );\n\n  return base.equals(head) ? null : `- \\`${file}\\` changed`;\n};\n\nconst files = [...new Set([...baseFiles, ...headFiles])].sort();\nconst changes = (await Promise.all(files.map(describe))).filter(Boolean);\n\nconst sections = [];\n\nif (changes.length) {\n  sections.push(\n    `**Output:** ${changes.length} ${changes.length === 1 ? 'file differs' : 'files differ'}`,\n    changes.join('\\n')\n  );\n}\n\nconst performance = await comparePerformance();\n\nif (performance) {\n  sections.push(performance);\n}\n\nif (sections.length) {\n  console.log(`${TITLE}\\n\\n${sections.join('\\n\\n')}\\n`);\n}\n```","summary":"This example lists every file whose bytes changed, which catches the same-size edits that `file-size` misses:","examples":[{"language":"mjs","displayName":"scripts/comparators/changed-files.mjs","code":"import { readFile } from 'node:fs/promises';\nimport { join } from 'node:path';\n\nimport { BASE, HEAD, TITLE } from '../constants.mjs';\nimport { listOutputFiles } from './files.mjs';\nimport { comparePerformance } from './performance.mjs';\n\nconst [baseFiles, headFiles] = await Promise.all(\n  [BASE, HEAD].map(directory => listOutputFiles(directory))\n);\nconst baseSet = new Set(baseFiles);\nconst headSet = new Set(headFiles);\n\nconst describe = async file => {\n  if (!baseSet.has(file)) {\n    return `- \\`${file}\\` added`;\n  }\n\n  if (!headSet.has(file)) {\n    return `- \\`${file}\\` removed`;\n  }\n\n  const [base, head] = await Promise.all(\n    [BASE, HEAD].map(directory => readFile(join(directory, file)))\n  );\n\n  return base.equals(head) ? null : `- \\`${file}\\` changed`;\n};\n\nconst files = [...new Set([...baseFiles, ...headFiles])].sort();\nconst changes = (await Promise.all(files.map(describe))).filter(Boolean);\n\nconst sections = [];\n\nif (changes.length) {\n  sections.push(\n    `**Output:** ${changes.length} ${changes.length === 1 ? 'file differs' : 'files differ'}`,\n    changes.join('\\n')\n  );\n}\n\nconst performance = await comparePerformance();\n\nif (performance) {\n  sections.push(performance);\n}\n\nif (sections.length) {\n  console.log(`${TITLE}\\n\\n${sections.join('\\n\\n')}\\n`);\n}"}],"children":[]},{"kind":"section","id":"run-it-locally","name":"Run it locally","title":"Run it locally","scope":"module","overloadOf":null,"stability":null,"added":[],"deprecated":[],"removed":[],"napiVersion":[],"changes":[],"description":"Build the same target from `main` into `base/` and from your branch into\n`out/`, with the same Node.js checkout for both (see [Running the tool\nlocally](https://github.com/nodejs/doc-kit/blob/main/CONTRIBUTING.md#running-the-tool-locally)\nfor a sparse checkout of the Node.js docs), then run the script:\n\n```bash\ngit switch main\nnode packages/cli/bin/cli.mjs generate -t man-page -i ../node/doc/api/cli.md -o base\n\ngit switch my-branch\nnode packages/cli/bin/cli.mjs generate -t man-page -i ../node/doc/api/cli.md -o out\n\nGENERATOR=man-page node scripts/comparators/changed-files.mjs\n```\n\nLocal builds have no `benchmark.json`, so the report has no performance\nsection.","summary":"Build the same target from `main` into `base/` and from your branch into `out/`, with the same Node.js checkout for both (see Running the tool locally for a sparse checkout of the Node.js docs), then run the script:","examples":[{"language":"bash","displayName":null,"code":"git switch main\nnode packages/cli/bin/cli.mjs generate -t man-page -i ../node/doc/api/cli.md -o base\n\ngit switch my-branch\nnode packages/cli/bin/cli.mjs generate -t man-page -i ../node/doc/api/cli.md -o out\n\nGENERATOR=man-page node scripts/comparators/changed-files.mjs"}],"children":[]},{"kind":"section","id":"add-a-test","name":"Add a test","title":"Add a test","scope":"module","overloadOf":null,"stability":null,"added":[],"deprecated":[],"removed":[],"napiVersion":[],"changes":[],"description":"Comparator tests live in `scripts/__tests__/comparators.test.mjs` and run with\n`node --run test`. The file's `createDirectories` helper creates temporary base\nand head directories, and `runComparator` runs a comparator against them as a\nchild process with `GENERATOR=test`:\n\n```mjs displayName=\"scripts/__tests__/comparators.test.mjs\"\ntest('changed-files comparator reports same-size edits', async t => {\n  const { base, head } = await createDirectories(t);\n\n  await Promise.all([\n    writeFile(path.join(base, 'node.1'), 'base', 'utf8'),\n    writeFile(path.join(head, 'node.1'), 'head', 'utf8'),\n  ]);\n\n  const result = await runComparator('changed-files', base, head);\n\n  assert.match(result, /## `test` Generator/);\n  assert.match(result, /`node\\.1` changed/);\n});\n```","summary":"Comparator tests live in `scripts/__tests__/comparators.test.mjs` and run with `node --run test`. The file's `createDirectories` helper creates temporary base and head directories, and `runComparator` runs a comparator against them as a child process with `GENERATOR=test`:","examples":[{"language":"mjs","displayName":"scripts/__tests__/comparators.test.mjs","code":"test('changed-files comparator reports same-size edits', async t => {\n  const { base, head } = await createDirectories(t);\n\n  await Promise.all([\n    writeFile(path.join(base, 'node.1'), 'base', 'utf8'),\n    writeFile(path.join(head, 'node.1'), 'head', 'utf8'),\n  ]);\n\n  const result = await runComparator('changed-files', base, head);\n\n  assert.match(result, /## `test` Generator/);\n  assert.match(result, /`node\\.1` changed/);\n});"}],"children":[]},{"kind":"section","id":"turn-it-on-in-ci","name":"Turn it on in CI","title":"Turn it on in CI","scope":"module","overloadOf":null,"stability":null,"added":[],"deprecated":[],"removed":[],"napiVersion":[],"changes":[],"description":"Set `compare` on the matrix entries in `.github/workflows/generate.yml` that\nshould use it. For example, to compare the man page:\n\n```yaml displayName=\".github/workflows/generate.yml\"\n- target: man-page\n  input: './node/doc/api/cli.md'\n  compare: changed-files\n```\n\nThe compare step runs in the pull request's own checkout, so the pull request\nthat adds a comparator already uses it. The base side is the target's artifact\nfrom the `main` run, which every matrix entry uploads whether or not it's\ncompared.\n\nA target that isn't in the matrix on `main` yet has no artifact there, so its\ndownload step fails. Add its matrix entry without `compare` first, and turn\ncomparison on in a later pull request.","summary":"Set `compare` on the matrix entries in `.github/workflows/generate.yml` that should use it. For example, to compare the man page:","examples":[{"language":"yaml","displayName":".github/workflows/generate.yml","code":"- target: man-page\n  input: './node/doc/api/cli.md'\n  compare: changed-files"}],"children":[]}]}]}