Creating generators

Every output of doc-kit is a generator, and yours can be one too: an RSS feed, a changelog, TypeScript declarations, anything the API entries can become. This guide starts with a generator in one file, then adds options, worker threads and streaming, and ends with publishing it as a package.

A generator is a module whose default export gives its name, the generator it reads from (dependsOn), and a generate function that receives that generator's output. This one lists every experimental API:

import { join } from 'node:path';

import getConfig from '@doc-kit/core/utils/configuration/index.mjs';
import { writeFile } from '@doc-kit/core/utils/file.mjs';

export default {
  name: 'experimental',
  description: 'Lists every experimental API',

  // The typed API entries, the same the site and the JSON are built from
  dependsOn: '@doc-kit/core/metadata',

  async generate(entries) {
    const { output } = getConfig('experimental');

    const list = entries
      .filter(entry => entry.stability?.data.index === '1')
      .map(entry => `- ${entry.heading.data.name} (${entry.api})`);

    await writeFile(join(output, 'experimental.md'), list.join('\n'));
  },
};

Pass its path as a target, next to any others. Here it runs next to the website (html), which comes from @doc-kit/generator-react, so install that package beside the CLI:

npx doc-kit generate -t html -t ./experimental.mjs -i "docs/**/*.md" -o out

It reads the same parse of the docs as html, and writes out/experimental.md. (With pnpm, add @doc-kit/core to your dependencies so the module can import it.)

Each generator declares the generator it reads from with dependsOn. doc-kit follows those links from the targets it's given, loads every generator they need, and runs the ones that don't depend on one another at the same time. The shared stages are documented with the rest of the generators.

Besides name, dependsOn and generate, a generator can declare default options, a function to run in worker threads, and a Markdown pipeline. The rest of this guide covers them one at a time, and @doc-kit/core lists every property in one place.

defaultConfiguration holds the defaults of the generator's options. Users override them in a section named after the generator in their configuration file, and getConfig(name) returns the result:

import { join } from 'node:path';

import getConfig from '@doc-kit/core/utils/configuration/index.mjs';
import { writeFile } from '@doc-kit/core/utils/file.mjs';

export default {
  name: 'experimental',
  description: 'Lists every experimental API',
  dependsOn: '@doc-kit/core/metadata',

  defaultConfiguration: {
    // The file the list is written to, in the output directory
    fileName: 'experimental.md',
  },

  async generate(entries) {
    const { output, fileName } = getConfig('experimental');

    const list = entries
      .filter(entry => entry.stability?.data.index === '1')
      .map(entry => `- ${entry.heading.data.name} (${entry.api})`);

    await writeFile(join(output, fileName), list.join('\n'));
  },
};

export default {
  experimental: {
    fileName: 'unstable.md',
  },
};

• getConfig(name) is the default export of @doc-kit/core/utils/configuration/index.mjs. Besides the generator's own options, what it returns holds every global option the generator doesn't set itself: output, project, minify, ref, and version (a SemVer from the semver package) among them.
• A default for a global option wins over the global value for that generator: jsx-ast defaults ref to 'main'. Users still change it in the generator's own section.
• defaultConfiguration can also be a function, which receives the user's configuration (the configuration file and command-line options, with targets resolved to import specifiers) and returns the defaults. The html generator uses one to show its search box only when the search index is also a target.
• getConfig() without a name returns the whole configuration on the main thread. Worker threads only get the generator's own section.

generate(input, worker) receives the complete output of the generator it depends on. When that generator streams, its chunks arrive collected in one array. These are the stages most generators read from:

To read from a generator of your own, point dependsOn at its module with import.meta.resolve('./other.mjs'), since a relative path resolves from the working directory. Without a dependsOn, generate receives undefined and reads its own sources, as ast does.

What generate returns, or resolves to, is what the generators depending on it receive. The CLI doesn't print or save a target's result, so a generator producing an output writes its files itself. Many do both:

import { join } from 'node:path';

import getConfig from '@doc-kit/core/utils/configuration/index.mjs';
import { writeJSON } from '@doc-kit/core/utils/file.mjs';

export default {
  name: 'experimental',
  description: 'Lists every experimental API',
  dependsOn: '@doc-kit/core/metadata',

  async generate(entries) {
    const { output, minify } = getConfig('experimental');

    const experimental = entries.filter(
      entry => entry.stability?.data.index === '1'
    );

    if (output) {
      await writeJSON(join(output, 'experimental.json'), experimental, minify);
    }

    // What the generators reading from this one receive
    return experimental;
  },
};

@doc-kit/core/utils/file.mjs has two helpers for writing:

• writeFile(file, data[, options]) writes a file as fs.promises.writeFile() does, creating its directory first.
• writeJSON(file, value[, minify]) writes a value as JSON, indented unless minify is true.

output is undefined when no output directory is given, which is why the built-in generators check it before writing. To copy static files that ship with your module, use cp() from node:fs/promises:

await cp(new URL('./assets', import.meta.url), join(output, 'assets'), {
  recursive: true,
});

For heavy work on many independent items, give the generator a processChunk and set hasParallelProcessor. generate then receives a worker, whose stream() splits the items into chunks and runs processChunk on them across worker threads:

import { join } from 'node:path';

import getConfig from '@doc-kit/core/utils/configuration/index.mjs';
import { writeFile } from '@doc-kit/core/utils/file.mjs';

export default {
  name: 'experimental',
  description: 'Lists every experimental API',
  dependsOn: '@doc-kit/core/metadata',

  hasParallelProcessor: true,

  // Runs in a worker thread, once per chunk of entries
  async processChunk(input, itemIndices, { marker }) {
    return itemIndices
      .map(index => input[index])
      .filter(entry => entry.stability?.data.index === '1')
      .map(entry => `${marker} ${entry.heading.data.name} (${entry.api})`);
  },

  async generate(entries, worker) {
    const { output } = getConfig('experimental');
    const lines = [];

    for await (const chunk of worker.stream(entries, { marker: '-' })) {
      lines.push(...chunk);
    }

    // Chunks arrive in the order they finish
    lines.sort();

    await writeFile(join(output, 'experimental.md'), lines.join('\n'));
  },
};

Filtering entries is light work, but heavier work takes the same shape.

• worker.stream(items, deps) splits items into chunks of chunkSize items (10 by default), runs them on up to threads workers, and yields each chunk's result as it finishes, in the order they finish. Every processChunk call gets deps.
• processChunk(input, itemIndices, deps) processes the items at itemIndices and resolves to an array of results. Read each item as input[index]: in a worker, input holds only the chunk's items, but with one thread, or two items or fewer, processChunk runs on the main thread with the whole input.
• The items, deps, the results, and the generator's configuration section are copied between threads with the structured clone algorithm. Functions can't be copied (passing one throws a DataCloneError), and class instances arrive as plain objects.
• Each worker imports your module on its own, so processChunk sees nothing generate set up: pass what it needs in deps. A module only processChunk uses can be imported inside it, as the metadata generator does, so the main thread never loads it.
• In a worker, getConfig(name) returns the generator's own section, with the global options it inherits, and nothing else. Its version is a plain object there, not a SemVer.

generate either resolves to its whole output, as in the examples above, or streams it: as an async generator function, it yields its results in arrays, usually each chunk as a worker finishes it.

export default {
  name: 'experimental',
  description: 'Lists every experimental API',
  dependsOn: '@doc-kit/core/metadata',

  hasParallelProcessor: true,

  async processChunk(input, itemIndices, { marker }) {
    return itemIndices
      .map(index => input[index])
      .filter(entry => entry.stability?.data.index === '1')
      .map(entry => `${marker} ${entry.heading.data.name} (${entry.api})`);
  },

  // Yields each chunk's lines as soon as a worker finishes them
  async *generate(entries, worker) {
    for await (const chunk of worker.stream(entries, { marker: '-' })) {
      yield chunk;
    }
  },
};

doc-kit collects what a streaming generator yields into one array, once, and hands that array to every generator reading from it:

• Each value it yields must be an array, whose items are appended to the output.
• The output keeps the order the values were yielded in, which for worker.stream() is the order the chunks finished in.
• The generators reading from it start when the stream ends, with the complete output.

Streaming suits a stage other generators build on, whose output is just its chunks' results, as ast, metadata, and jsx-ast are. Collect when the generator needs every result before it can finish: to sort them, cross-reference them, or write them to a single file.

dependsOn pulls another generator's output in. The inverse, dependent, pushes a generator's output into another generator's pipeline. Use it when a generator transforms an intermediate representation (adding, filtering, or rewriting entries) rather than producing an output of its own. The section-pages generator, which adds a page for each section of a module, declares two:

export default {
  name: 'section-pages',

  dependsOn: '@doc-kit/core/metadata',

  // Deliver this generator's output through `html` and `sitemap`
  dependent: [
    '@doc-kit/generator-react/html',
    '@doc-kit/generator-react/sitemap',
  ],

  generate,
};

doc-kit walks up each dependent's pipeline, starting at the dependent itself, and splices the generator in front of the first generator there that reads the same dependsOn. html depends on jsx-ast, which depends on metadata, so jsx-ast is rewired to read from section-pages instead. sitemap reads metadata directly, so it's rewired itself.

• Requesting a generator that declares dependents delivers its output through them: through the ones the run already includes, or through all of them when it includes none. -t section-pages builds the site and the sitemap, while -t html -t section-pages builds just the site.
• Several generators can splice in at the same point; they form a chain.
• A dependent whose pipeline never reads the generator's dependsOn is an error.

A generator processing Markdown declares its unified pipeline as markdown, and gets a processor for it with getProcessor(name):

import { join } from 'node:path';

import getConfig from '@doc-kit/core/utils/configuration/index.mjs';
import { writeFile } from '@doc-kit/core/utils/file.mjs';
import { getProcessor } from '@doc-kit/core/utils/markdown/processor.mjs';

export default {
  name: 'entries-html',
  description: 'Renders every API entry to HTML',
  dependsOn: '@doc-kit/core/metadata',

  markdown: {
    remarkPlugins: ['...'],
    rehypePlugins: [
      ['remark-rehype', { allowDangerousHtml: true }],
      '...',
      '@doc-kit/core/plugins/shiki/rehype.mjs',
      ['rehype-stringify', { allowDangerousHtml: true }],
    ],
  },

  async generate(entries) {
    const { output } = getConfig('entries-html');
    const processor = getProcessor('entries-html');

    const html = await Promise.all(
      entries.map(async ({ content }) =>
        processor.stringify(await processor.run(content))
      )
    );

    await writeFile(join(output, 'entries.html'), html.join('\n'));
  },
};

Plugins are listed as in the markdown option, and resolve from the generator's module: paths relative to it, and packages it can import (this one needs remark-rehype and rehype-stringify installed).

• The lists run in order, on one processor: remark plugins on the Markdown tree, rehype plugins on HTML, and recma plugins on JavaScript. So a rehype list starts with remark-rehype, and a recma list with rehype-recma.
• '...' marks where a list takes the configured plugins, from global and from the generator's own section. A list without one takes none, and doc-kit warns about plugins configured for it. A generator rendering Markdown (one with rehype or recma plugins) only takes the remark plugins of its own section, as the global ones already ran when ast parsed the files.
• Each thread imports the generator, and its pipeline with it, so the options in markdown can be anything, functions included. getProcessor() works in generate and in processChunk alike.
• A plugin's module default-exports the plugin, a list of plugins, or a preset. A plugin that needs asynchronous setup exports an async load(options) returning the plugin instead.

getProcessor(name[, options]) takes these options:

The Shiki plugin, @doc-kit/core/plugins/shiki/rehype.mjs, highlights code blocks in every language Shiki bundles; its options are described in Code highlighting. To highlight other code the way the pipeline does, getHighlighter(name) returns the highlighter of a generator's Shiki plugin, and throws when its pipeline has none:

import { getHighlighter } from '@doc-kit/core/plugins/shiki/highlighter.mjs';

// In `generate` or `processChunk`
const highlighter = getHighlighter('entries-html');

const html = highlighter.highlightToHtml('const answer = 42;', 'js');

@doc-kit/core declares the types of a generator as globals, which importing its generators/types module brings in:

import type {} from '@doc-kit/core/generators/types';
import type { MetadataEntry } from '@doc-kit/core/generators/metadata/types';
import type { GlobalConfiguration } from '@doc-kit/core/utils/configuration/types';

export type Configuration = {
  fileName: string;
};

// What `getConfig('experimental')` returns
export type Options = GlobalConfiguration & Configuration;

export type Generator = Omit<
  GeneratorMetadata<
    // Its options
    Configuration,
    // What `generate` receives, and what it resolves to
    Generate<Array<MetadataEntry>, void>,
    // What `processChunk` gets per item, what it resolves to, and its `deps`
    ProcessChunk<MetadataEntry, Array<string>, { marker: string }>
  >,
  // `GeneratorMetadata` only takes the names of the built-in generators
  'name'
> & { name: string };

import getConfig from '@doc-kit/core/utils/configuration/index.mjs';

/** @type {import('./types').Generator} */
export default {
  name: 'experimental',
  description: 'Lists every experimental API',
  dependsOn: '@doc-kit/core/metadata',
  defaultConfiguration: { fileName: 'experimental.md' },

  async generate(entries) {
    /** @type {import('./types').Options} */
    // @ts-expect-error `getConfig()` only knows the built-in generators
    const { output, fileName } = getConfig('experimental');

    // ...
  },
};

• The type requires description, dependsOn and defaultConfiguration, even though doc-kit doesn't: give a generator without options defaultConfiguration: {}, and one that reads its own sources dependsOn: undefined.
• getConfig() is typed for the built-in names only, so type checking flags a call with your generator's name. Mark that call with // @ts-expect-error and type its result yourself, as above.
• A generator that streams declares AsyncGenerator<Array<Item>> as what generate returns.

A package exports each of its generators from a subpath:

{
  "name": "doc-kit-rss",
  "version": "1.0.0",
  "type": "module",
  "exports": {
    "./rss": "./src/rss/index.mjs"
  },
  "peerDependencies": {
    "@doc-kit/core": "^1.3.0"
  }
}

Users install it next to the CLI and pass the subpath as a target, on the command line or in the target of their configuration file:

npm install --save-dev @doc-kit/cli doc-kit-rss
npx doc-kit generate -t doc-kit-rss/rss -i "docs/**/*.md" -o out

• List @doc-kit/core as a peer dependency, so the generator shares the CLI's copy: getConfig() reads the configuration the CLI gave that copy.
• A generator it depends on from another package, such as @doc-kit/generator-react/jsx-ast, needs that package installed too.
• Its name is the section users configure it in, so pick one that doesn't collide with the built-in generators.

In this repository, each generator has a directory of its own, with the object in index.mjs, its functions in generate.mjs and its types in types.d.ts. A new generator also gets a subpath in its package's exports, such as "./my-format": "./src/generators/my-format/index.mjs" in packages/core/package.json, and a shorthand name in packages/core/src/generators/index.mjs, which maps the name users type to that specifier. The key matches the generator's name:

export const publicGenerators = {
  json: '@doc-kit/core/json',
  'my-format': '@doc-kit/core/my-format',
  // ...
};

publicGenerators are the targets doc-kit generate --help lists. Stages that produce no output of their own go in internalGenerators instead, which the help leaves out. A README.md next to the generator's index.mjs, documenting its options, becomes its page on this site; add it to the sidebar in www/doc-kit.config.mjs. The contributing guide covers the rest, from running the tests to adding a changeset.