en Async Interpret - chiba233/yume-dsl-token-walker GitHub Wiki

Async Interpret API

Interpret (sync) | Exports | Home

The async API is a full mirror of the synchronous Interpret API. Same semantics, same safety guarantees — but interpret can await, and interpretChildren returns AsyncIterable.

When to use instead of sync:

  • Your handler needs to fetch remote content
  • Your handler queries a database
  • Your handler calls an async renderer (e.g. Shiki codeToHtml)

If none of your handlers are async, use the sync API — it's simpler and has zero async overhead.

Quick Start

import { createParser, createSimpleInlineHandlers } from "yume-dsl-rich-text";
import { interpretTextAsync, collectNodesAsync } from "yume-dsl-token-walker";

const parser = createParser({
  handlers: createSimpleInlineHandlers(["bold"]),
});

const html = (
  await collectNodesAsync(
    interpretTextAsync("Hello $$bold(world)$$", parser, {
      createText: (text) => text,
      interpret: async (token, helpers) => {
        if (token.type === "bold") {
          return {
            type: "nodes",
            nodes: (async function* () {
              yield "<b>";
              yield* helpers.interpretChildren(token.value);
              yield "</b>";
            })(),
          };
        }
        return { type: "unhandled" };
      },
    }, undefined),
  )
).join("");

// → "Hello <b>world</b>"

Demo: fetch remote content in a handler

import type { AsyncInterpretRuleset } from "yume-dsl-token-walker";
import { interpretTokensAsync, collectNodesAsync } from "yume-dsl-token-walker";

const ruleset: AsyncInterpretRuleset<string, void> = {
  createText: (text) => text,
  interpret: async (token, helpers) => {
    if (token.type === "embed") {
      const url = typeof token.url === "string" ? token.url : "";
      const response = await fetch(url);
      const html = await response.text();
      return { type: "text", text: `<div class="embed">${html}</div>` };
    }
    if (token.type === "bold") {
      return {
        type: "nodes",
        nodes: (async function* () {
          yield "<b>";
          yield* helpers.interpretChildren(token.value);
          yield "</b>";
        })(),
      };
    }
    return { type: "unhandled" };
  },
  onUnhandled: "flatten",
};

const nodes = await collectNodesAsync(
  interpretTokensAsync(tokens, ruleset, undefined),
);

Key design differences from sync

Aspect Sync Async
interpret return InterpretResult<TNode> Awaitable<AsyncInterpretResult<TNode>> — can return a Promise
interpretChildren return Iterable<TNode> AsyncIterable<TNode> — consume with for await or yield*
nodes in result Iterable<TNode> Iterable<TNode> | AsyncIterable<TNode> — plain arrays and async generators both work
createText Synchronous Still synchronous — text wrapping is a pure operation
onUnhandled function Returns ResolvedResult Returns Awaitable<AsyncResolvedResult>
Error handling, recursion detection Same Same

Async helpers

Helper Description
fromAsyncHandlerMap(handlers) Build an async interpret function from Record<type, asyncHandler>. Unmatched tokens return { type: "unhandled" }
wrapAsyncHandlers(handlers, wrap) Wrap every async handler with shared logic. The wrap callback receives the awaited result
collectNodesAsync(iterable) Collect an AsyncIterable<TNode> into an array. Equivalent to a for await loop

Type reference

interpretTextAsync

async function* interpretTextAsync<TNode, TEnv>(
  input: string,
  parser: ParserLike,
  ruleset: AsyncInterpretRuleset<TNode, TEnv>,
  env: TEnv,
): AsyncGenerator<TNode>;

interpretTokensAsync

async function* interpretTokensAsync<TNode, TEnv>(
  tokens: TextToken[],
  ruleset: AsyncInterpretRuleset<TNode, TEnv>,
  env: TEnv,
): AsyncGenerator<TNode>;

AsyncInterpretRuleset

interface AsyncInterpretRuleset<TNode, TEnv = unknown> {
  createText: (text: string) => TNode;                    // always sync
  interpret: (
    token: TextToken,
    helpers: AsyncInterpretHelpers<TNode, TEnv>,
  ) => Awaitable<AsyncInterpretResult<TNode>>;
  onUnhandled?: AsyncUnhandledStrategy<TNode, TEnv>;      // default: "flatten"
  onError?: (context: {
    error: Error;
    phase: "interpret" | "flatten" | "traversal" | "internal";
    token?: TextToken;
    position?: SourceSpan;
    env: TEnv;
  }) => void;
}

AsyncInterpretHelpers

interface AsyncInterpretHelpers<TNode, TEnv = unknown> {
  interpretChildren: (value: string | TextToken[]) => AsyncIterable<TNode>;
  flattenText: (value: string | TextToken[]) => string;   // still sync
  env: TEnv;
}

AsyncInterpretResult

type AsyncInterpretResult<TNode> =
  | { type: "nodes"; nodes: Iterable<TNode> | AsyncIterable<TNode> }
  | { type: "text"; text: string }
  | { type: "flatten" }
  | { type: "unhandled" }
  | { type: "drop" };

AsyncUnhandledStrategy

type AsyncUnhandledStrategy<TNode, TEnv = unknown> =
  | "throw"
  | "flatten"
  | "drop"
  | ((
      token: TextToken,
      helpers: AsyncInterpretHelpers<TNode, TEnv>,
    ) => Awaitable<AsyncResolvedResult<TNode>>);

Awaitable

type Awaitable<T> = T | Promise<T>;

Used throughout async API signatures so you can return either a plain value or a Promise.

⚠️ **GitHub.com Fallback** ⚠️