Adding a comparator

A comparator is a small Node.js script that compares one generator's output on main with its output on a pull request and prints a Markdown report. CI collects these reports into a comment on the pull request, so you and the reviewers can see what a change does to the generated docs, and to generation time and memory, before it lands.

Comparators live in scripts/comparators/ and run only in this repository's CI. They are not part of any published package.

Two workflows work together:

1. .github/workflows/generate.yml ("Generate and Compare Docs") runs on every push to main. It records the latest nodejs/node commit, generates every target in its matrix from that commit's docs, and uploads each target's output directory as an artifact named after the target.
2. On a pull request, the same workflow finds the main run for the pull request's base commit and reuses its Node.js commit. Both sides are built from the same sources, so any difference comes from the pull request.
3. Each matrix entry generates into out/. Those with a compare field then download the target's artifact from the main run into base/ and run node scripts/comparators/<compare>.mjs > out/comparison.txt.
4. When the workflow finishes, .github/workflows/leave-comment.yml joins every comparison.txt and posts the result as a single comment on the pull request (tagged compared, so later pushes update it). If every report is empty, it posts nothing.

Each generation runs under /usr/bin/time, which writes out/benchmark.json with the elapsed time, user and system CPU time, and peak resident memory. Comparators read it for their performance section and leave it out of the output comparison.

Matrix entries without compare (at the moment man-page, addon-verify and json-simple) are generated and uploaded, but not compared.

Both end with a Performance estimate section: the generation time and peak memory of each side, read from benchmark.json. It comes from a single CI run, so treat it as a rough signal. It's left out when either side has no benchmark.json.

Keep their limits in mind when you pick one:

• file-size compares sizes only. An edit that keeps a file's size the same doesn't show up.
• object-assertion parses every output file as JSON, so it only suits generators whose output is all JSON.

A comparator is an ES module in scripts/comparators/. Its file name without .mjs is the value of the compare field in the workflow matrix. It should:

• Read the two builds from BASE and HEAD, listing their files with listOutputFiles, which skips benchmark.json and the comparison.txt that the report is being written to.
• Print Markdown to standard output, starting with TITLE, and print nothing when there's nothing to report.
• Report differences instead of failing on them. A comparator that exits with an error fails the CI job, so throw only when something is actually broken.
• End with comparePerformance(), like the built-in comparators, so the timing stays visible.

This example lists every file whose bytes changed, which catches the same-size edits that file-size misses:

import { readFile } from 'node:fs/promises';
import { join } from 'node:path';

import { BASE, HEAD, TITLE } from '../constants.mjs';
import { listOutputFiles } from './files.mjs';
import { comparePerformance } from './performance.mjs';

const [baseFiles, headFiles] = await Promise.all(
  [BASE, HEAD].map(directory => listOutputFiles(directory))
);
const baseSet = new Set(baseFiles);
const headSet = new Set(headFiles);

const describe = async file => {
  if (!baseSet.has(file)) {
    return `- \`${file}\` added`;
  }

  if (!headSet.has(file)) {
    return `- \`${file}\` removed`;
  }

  const [base, head] = await Promise.all(
    [BASE, HEAD].map(directory => readFile(join(directory, file)))
  );

  return base.equals(head) ? null : `- \`${file}\` changed`;
};

const files = [...new Set([...baseFiles, ...headFiles])].sort();
const changes = (await Promise.all(files.map(describe))).filter(Boolean);

const sections = [];

if (changes.length) {
  sections.push(
    `**Output:** ${changes.length} ${changes.length === 1 ? 'file differs' : 'files differ'}`,
    changes.join('\n')
  );
}

const performance = await comparePerformance();

if (performance) {
  sections.push(performance);
}

if (sections.length) {
  console.log(`${TITLE}\n\n${sections.join('\n\n')}\n`);
}

Build the same target from main into base/ and from your branch into out/, with the same Node.js checkout for both (see Running the tool locally for a sparse checkout of the Node.js docs), then run the script:

git switch main
node packages/cli/bin/cli.mjs generate -t man-page -i ../node/doc/api/cli.md -o base

git switch my-branch
node packages/cli/bin/cli.mjs generate -t man-page -i ../node/doc/api/cli.md -o out

GENERATOR=man-page node scripts/comparators/changed-files.mjs

Local builds have no benchmark.json, so the report has no performance section.

Comparator tests live in scripts/__tests__/comparators.test.mjs and run with node --run test. The file's createDirectories helper creates temporary base and head directories, and runComparator runs a comparator against them as a child process with GENERATOR=test:

test('changed-files comparator reports same-size edits', async t => {
  const { base, head } = await createDirectories(t);

  await Promise.all([
    writeFile(path.join(base, 'node.1'), 'base', 'utf8'),
    writeFile(path.join(head, 'node.1'), 'head', 'utf8'),
  ]);

  const result = await runComparator('changed-files', base, head);

  assert.match(result, /## `test` Generator/);
  assert.match(result, /`node\.1` changed/);
});

Set compare on the matrix entries in .github/workflows/generate.yml that should use it. For example, to compare the man page:

- target: man-page
  input: './node/doc/api/cli.md'
  compare: changed-files

The compare step runs in the pull request's own checkout, so the pull request that adds a comparator already uses it. The base side is the target's artifact from the main run, which every matrix entry uploads whether or not it's compared.

A target that isn't in the matrix on main yet has no artifact there, so its download step fails. Add its matrix entry without compare first, and turn comparison on in a later pull request.