en Structural Slice - chiba233/yume-dsl-token-walker GitHub Wiki

Structural Slice

Lint | Exports | Home

Re-parse only a region of a large document instead of the whole thing. parseStructural gives you the map; parseSlice lets you jump to any point on it and get fully positioned TextToken[].

When to use:

  • An editor user edits one tag in a 200 KB document — only that tag needs re-parsing
  • You want to render a specific region without paying for a full parse
  • You need TextToken[] with positions that point back to the original document

When implicit inline shorthand is enabled upstream, parseSlice is shorthand-aware: if direct span parse degenerates to plain-text echo on an implicit shorthand inline span, it reparses the enclosing parent tag span (when available).

Demo: incremental editing pipeline

The full pipeline: structural scan → find the edited region → re-parse only that region → interpret:

import { createParser, createSimpleInlineHandlers, buildPositionTracker } from "yume-dsl-rich-text";
import { parseSlice, interpretTokens, collectNodes } from "yume-dsl-token-walker";

const parser = createParser({
  handlers: createSimpleInlineHandlers(["bold", "italic"]),
});

const fullText = "intro\n$$bold(hello $$italic(world)$$)$$\noutro";

// 1. Pre-scan: fast structural pass with positions
const structural = parser.structural(fullText, { trackPositions: true });

// 2. Build tracker once — reuse for all slices
const tracker = buildPositionTracker(fullText);

// 3. Pick a node and parse just that region
const boldNode = structural.find((n) => n.type === "inline" && n.tag === "bold");

if (boldNode?.position) {
  const tokens = parseSlice(fullText, boldNode.position, parser, tracker);
  // tokens have correct offset/line/column relative to fullText

  // 4. Interpret as usual
  const html = collectNodes(
    interpretTokens(tokens, {
      createText: (t) => t,
      interpret: (token, helpers) => {
        if (token.type === "bold")
          return { type: "nodes", nodes: ["<b>", ...helpers.interpretChildren(token.value), "</b>"] };
        if (token.type === "italic")
          return { type: "nodes", nodes: ["<em>", ...helpers.interpretChildren(token.value), "</em>"] };
        return { type: "unhandled" };
      },
    }, undefined),
  ).join("");
}

With vs. without tracker

Without tracker With tracker
offset Correct (shifted by baseOffset) Correct
line Local to the slice (starts at 1) Correct — points to original document
column Local to the slice Correct — points to original document

Build the tracker once with buildPositionTracker(fullText). Do not rebuild per slice — it only needs rebuilding when newlines in the document change (i.e., lines are inserted or deleted).

Performance

Measured on a ~200 KB document (210K chars, 2562 tokens, 1281 real tag nodes). Environment: Kunpeng 920 aarch64 / Node v24.14.0 — 3 rounds x 5 iterations, averaged.

Step Time Notes
Full parseRichText ~24 ms Full handler pipeline on 200 KB
Full parseStructural + tracking ~31 ms Comparable to parseRichText; structural-only scan
nodeAtOffset (locate) ~0.14 ms Traverse cached structural tree
parseSlice (incremental) ~0.025 ms Parse only the 36-char edited node
buildPositionTracker (rebuild) ~1.06 ms Only needed when newlines change

Incremental path (locate + slice) = ~0.17 ms — only reparses the touched region. Full parseRichText is already fast (~24 ms for 200 KB); parseSlice is for keystroke-level real-time editing where you want to reparse only the edited node, not the entire document. parseSlice cost scales with the slice size, not the document size.

API reference

parseSlice(fullText, span, parser, tracker?, fullTree?)

const parseSlice: (
  fullText: string,
  span: SourceSpan,
  parser: ParserLike,
  tracker?: PositionTracker,
  fullTree?: StructuralNode[],
) => TextToken[];
Param Description
fullText The complete source text
span Region to parse — typically StructuralNode.position
parser A parser with parse(input, overrides?)
tracker Optional tracker from buildPositionTracker(fullText) for correct line/column
fullTree Optional precomputed structural tree for shorthand fallback optimization

Position tracking is always enabled. baseOffset is derived from span.start.offset.

ParserLike

interface ParserLike {
  parse: (input: string, overrides?: ParseOverrides) => TextToken[];
  structural?: (input: string, overrides?: ParseOverrides) => StructuralNode[];
}

createParser(...) from yume-dsl-rich-text satisfies this interface.

ParseOverrides

interface ParseOverrides {
  trackPositions?: boolean;
  baseOffset?: number;
  tracker?: PositionTracker;
}
⚠️ **GitHub.com Fallback** ⚠️