{"$schema":"https://doc-kit.nodejs.org/schemas/api-doc/1.0.1.json","id":"generators-ast","path":"/generators/ast","type":"module","module":"generators-ast","title":"Parsing Markdown","introducedIn":null,"sourceLink":null,"stability":null,"added":[],"deprecated":[],"removed":[],"napiVersion":[],"changes":[],"description":"The `ast` stage reads your Markdown and MDX files and parses each one into a\nsyntax tree. Every output built from Markdown starts here, and it's where\nglobal remark plugins run, so configure it to change which files are read or\nhow they're parsed.\n\n**Stage:** `ast` · **Package:** [`@doc-kit/core`](https://doc-kit.nodejs.org/packages/core) · **Reads:** the files in `input` · **Writes:** no files","summary":"The `ast` stage reads your Markdown and MDX files and parses each one into a syntax tree. Every output built from Markdown starts here, and it's where global remark plugins run, so configure it to change which files are read or how they're parsed.","examples":[],"children":[{"kind":"section","id":"usage","name":"Usage","title":"Usage","scope":"module","overloadOf":null,"stability":null,"added":[],"deprecated":[],"removed":[],"napiVersion":[],"changes":[],"description":"Any target built from Markdown runs `ast` first, once per run, so you rarely\nname it. On its own it parses your inputs and writes nothing, which is a quick\nway to check that your MDX pages parse and your remark plugins load:\n\n```bash\nnpx doc-kit generate -t ast -i \"docs/**/*.{md,mdx}\"\n```\n\nA file that doesn't parse, such as an MDX page with an unclosed tag, stops the\nrun with the parser's error. The trees then go to the\n[API entries](https://doc-kit.nodejs.org/generators/metadata) stage.","summary":"Any target built from Markdown runs `ast` first, once per run, so you rarely name it. On its own it parses your inputs and writes nothing, which is a quick way to check that your MDX pages parse and your remark plugins load:","examples":[{"language":"bash","displayName":null,"code":"npx doc-kit generate -t ast -i \"docs/**/*.{md,mdx}\""}],"children":[]},{"kind":"section","id":"configuring","name":"Configuring","title":"Configuring","scope":"module","overloadOf":null,"stability":null,"added":[],"deprecated":[],"removed":[],"napiVersion":[],"changes":[],"description":"Options go in the `ast` section of the\n[configuration file](https://doc-kit.nodejs.org/configuration), for when this\nstage should read different files than the others:\n\n```mjs displayName=\"doc-kit.config.mjs\"\nexport default {\n  ast: {\n    input: ['docs/**/*.{md,mdx}'],\n    ignore: ['docs/drafts/**'],\n  },\n};\n```\n\n* `input` {string[]} Glob patterns of the files to parse, relative to the\n  working directory. It must be an array, even for a single pattern. Every\n  file it matches is parsed as Markdown or MDX, whatever its extension.\n  **Default:** from `global.input` (`--input`).\n* `ignore` {string | string[]} Glob patterns of files to leave out.\n  **Default:** from `global.ignore` (`--ignore`).\n* `markdown` {Object} [Markdown plugins](https://doc-kit.nodejs.org/configuration#markdown-plugins)\n  for this stage, run after the ones in `global.markdown`.\n  * `remarkPlugins` {Array} remark plugins, each a module specifier or a\n    `[specifier, options]` pair.\n\n`ast` never turns Markdown into HTML, so `rehypePlugins` and `recmaPlugins`\nlisted in `ast.markdown` are ignored with a warning. Once `ast` has an `input`\nof its own, `--input` no longer changes which Markdown files are parsed.\n\nSince every file `input` matches is parsed as Markdown, keep `.js` files out of\n`global.input` when a run also builds\n[API links](https://doc-kit.nodejs.org/generators/api-links): give them to the\n[Parsing JavaScript](https://doc-kit.nodejs.org/generators/ast-js) stage\nthrough its own `input` instead.","summary":"Options go in the `ast` section of the configuration file, for when this stage should read different files than the others:","examples":[{"language":"mjs","displayName":"doc-kit.config.mjs","code":"export default {\n  ast: {\n    input: ['docs/**/*.{md,mdx}'],\n    ignore: ['docs/drafts/**'],\n  },\n};"}],"children":[]},{"kind":"section","id":"what-happens-to-each-file","name":"What happens to each file","title":"What happens to each file","scope":"module","overloadOf":null,"stability":null,"added":[],"deprecated":[],"removed":[],"napiVersion":[],"changes":[],"description":"1. **Its syntax is picked.** Files ending in `.mdx` are parsed as MDX, all\n   others as Markdown. An `mdx: true` or `mdx: false` key in the file's\n   frontmatter overrides the extension (see\n   [MDX pages](https://doc-kit.nodejs.org/writing-docs#mdx-pages)). When the\n   frontmatter isn't valid YAML, the extension decides.\n2. **Its frontmatter becomes a metadata comment.** A `---` block at the very\n   start of the file is rewritten as a `<!-- YAML -->` comment, which the API\n   entries stage reads like one written by hand (see\n   [Page metadata](https://doc-kit.nodejs.org/writing-docs#page-metadata)). In\n   an MDX file, where HTML comments aren't valid, the block is removed before\n   parsing and the comment added back as the tree's first node.\n3. **Its first `Stability: N` becomes a link** to\n   `documentation.html#stability-index`, the section of the Node.js docs that\n   explains stability levels. On another site it only resolves when a\n   `documentation.html` with a \"Stability index\" heading sits next to the\n   page. The website's stability boxes show the level without the link, but\n   outputs that keep the Markdown, such as `llms-full.txt`, keep it.\n4. **It's parsed and run through the remark plugins** below.\n5. **It gets its page path** (see [Page paths](#page-paths)).","summary":"","examples":[],"children":[]},{"kind":"section","id":"the-plugin-pipeline","name":"The plugin pipeline","title":"The plugin pipeline","scope":"module","overloadOf":null,"stability":null,"added":[],"deprecated":[],"removed":[],"napiVersion":[],"changes":[],"description":"Each Markdown file goes through these remark plugins, in order:\n\n1. [`remark-parse`](https://github.com/remarkjs/remark/tree/main/packages/remark-parse),\n   which parses the Markdown.\n2. doc-kit's\n   [type annotations](https://doc-kit.nodejs.org/writing-docs#types-that-link-themselves),\n   which read text in curly braces, such as `{string}`, as a TypeScript type.\n3. [`remark-gfm`](https://github.com/remarkjs/remark-gfm), for GitHub Flavored\n   Markdown: tables, task lists, strikethrough, autolinks and footnotes.\n4. The plugins in `global.markdown.remarkPlugins`.\n5. The plugins in `ast.markdown.remarkPlugins`.\n\nMDX files add [`remark-mdx`](https://mdxjs.com/packages/remark-mdx/) before\nthese and skip the type annotations, since curly braces are JavaScript\nexpressions in MDX. A plugin can tell which kind of file it's processing from\n`this.data('mdx')`.\n\nEach plugin runs on the whole document and receives its file, whose\n`file.path` is the path `input` matched, such as `docs/guide/install.md`.\nBecause global remark plugins run here, every output sees their changes, and\n`jsx-ast`, which builds the website's pages, doesn't run them again.\n[Where plugins run](https://doc-kit.nodejs.org/configuration#where-plugins-run)\nlists the other generators that load them.","summary":"Each Markdown file goes through these remark plugins, in order:","examples":[],"children":[]},{"kind":"section","id":"page-paths","name":"Page paths","title":"Page paths","scope":"module","overloadOf":null,"stability":null,"added":[],"deprecated":[],"removed":[],"napiVersion":[],"changes":[],"description":"A file's page path is its path relative to the base of the pattern that\nmatched it (the directories before the first wildcard), without its extension.\nWith `docs/**/*.md`, `docs/guide/install.md` becomes `/guide/install`, which\nthe [website](https://doc-kit.nodejs.org/generators/html) writes as\n`guide/install.html`.\n\nEach pattern is matched on its own, so keep them from overlapping:\n\n* A file that two patterns match is parsed twice, and appears twice in every\n  output.\n* Patterns with different bases can give two files the same path:\n  `docs/index.md` and `api/index.md` are both `/index`.","summary":"A file's page path is its path relative to the base of the pattern that matched it (the directories before the first wildcard), without its extension. With `docs/**/*.md`, `docs/guide/install.md` becomes `/guide/install`, which the website writes as `guide/install.html`.","examples":[],"children":[]},{"kind":"section","id":"worker-threads","name":"Worker threads","title":"Worker threads","scope":"module","overloadOf":null,"stability":null,"added":[],"deprecated":[],"removed":[],"napiVersion":[],"changes":[],"description":"`ast` parses files in chunks of `chunkSize` files, spread across `threads`\nworker threads (see\n[Execution options](https://doc-kit.nodejs.org/configuration#execution-options)).\nWith `threads: 1`, or with two files or fewer, it parses them on the main\nthread instead. Each worker imports the remark plugins itself, which is why\nplugins are listed as module specifiers rather than imported functions.","summary":"`ast` parses files in chunks of `chunkSize` files, spread across `threads` worker threads (see Execution options). With `threads: 1`, or with two files or fewer, it parses them on the main thread instead. Each worker imports the remark plugins itself, which is why plugins are listed as module specifiers rather than imported functions.","examples":[],"children":[]},{"kind":"section","id":"what-it-produces","name":"What it produces","title":"What it produces","scope":"module","overloadOf":null,"stability":null,"added":[],"deprecated":[],"removed":[],"napiVersion":[],"changes":[],"description":"A [generator of your own](https://doc-kit.nodejs.org/creating-generators#input-and-output)\nwith `dependsOn: '@doc-kit/core/ast'` receives every file in one array, each\nas an object with:\n\n* `tree` {Object} The file's [mdast](https://github.com/syntax-tree/mdast)\n  syntax tree (a `Root` node), after every plugin ran.\n* `path` {string} Its page path, such as `/guide/install`.\n* `mdx` {boolean} Whether it was parsed as MDX.\n\nFiles arrive in the order their chunks finish, not the order of `input`. Most\ngenerators depend on `@doc-kit/core/metadata` instead, to get the API entries\nbuilt from these trees.","summary":"A generator of your own with `dependsOn: '@doc-kit/core/ast'` receives every file in one array, each as an object with:","examples":[],"children":[]}]}