Performance | API Reference | Source Position Tracking

parseStructural(...) is already fast, but if you're building an editor, you really don't want to re-scan the entire document on every keystroke. This page covers the incremental structural cache API in yume-dsl-rich-text — it lets you reparse only the part that changed and reuse everything else.

createIncrementalSession(...) and parseIncremental(...) are stable public APIs. Incremental heuristics and performance strategies may continue to evolve, but the session-first contract, result fields, and full-rebuild fallback semantics are part of the stable integration surface. You should still pin versions in production and keep a verified full-rebuild fallback path.

Under the hood, the incremental API maintains:

• tree: a StructuralNode[] parsed with trackPositions: true
• zones: a Zone[] built from that tree via buildZones(...)

It doesn't try to do tree-sitter style node reuse. Instead, it reparses a dirty range and reuses the untouched zones on either side.

• You're re-running structural parsing per keystroke on large documents (tens of KB+).
• You need a stable StructuralNode[] + Zone[] cache to power highlighting, outline, lint, and incremental pipelines.
• You want to pair structural caching with yume-dsl-token-walker's parseSlice(...) for re-parsing only the touched region into TextToken[].

If you only need substring parsing of a known region, parseSlice(...) alone is often enough.

Copy this:

import {createIncrementalSession} from "yume-dsl-rich-text";

const handlers = {/* ... */};
const session = createIncrementalSession(initialSource, {handlers});

// on each edit
const result = session.applyEdit(
    {startOffset, oldEndOffset, newText},
    newSource,
);
render(result.doc.tree);

If you need partial refresh info, switch to applyEditWithDiff:

const result = session.applyEditWithDiff(
    {startOffset, oldEndOffset, newText},
    newSource,
    undefined,
    {maxMilliseconds: 4},
);
refreshChangedPanels(result.diff);
render(result.doc.tree);

If you don't have newSource, build it yourself:

const previous = session.getDocument();
const newSource =
    previous.source.slice(0, startOffset) +
    newText +
    previous.source.slice(oldEndOffset);

As of 1.2.3, the low-level updater exports have been removed from the public surface — use the session API instead.

The whole API has just two roles:

Takes the full source, runs one structural parse, and returns a snapshot object doc (containing source / tree / zones).

It does not manage future edits — it only builds the starting state. Most of the time you don't need to call it directly; createIncrementalSession does it internally. Only call it when you explicitly want to own the initial snapshot as standalone data.

Returns a closure-based state container. It keeps one mutable currentDoc internally (the latest source / tree / zones) and exposes four methods:

Both applyEdit and applyEditWithDiff really advance the session state. The latter is not a "peek without committing" API. If you want to inspect a diff without mutating your real session, do it on a separate temporary session.

                       "I'm integrating incremental parsing"
                                    │
                  ┌─────────────────┼─────────────────┐
                  ▼                 ▼                  ▼
           First open?        Normal edit?        Don't trust cache /
                                    │             big config change?
                  │          ┌──────┴───────┐          │
                  ▼          ▼              ▼          ▼
        createIncremental   Need to know   Don't     session
        Session(...)        what changed?  need diff  .rebuild(...)
                                 │          │
                                 ▼          ▼
                          applyEdit      applyEdit
                          WithDiff         (...)
                            (...)

• edit tells the session which region of the old document changed — used to locate the dirty range for incremental work.
• newSource tells the session what the full document looks like after the edit — used for consistency validation and correctness.

Neither alone is sufficient: without newSource the session can't verify the final result; without edit it can't know where to focus.

import {
    createIncrementalSession,
    parseIncremental,
} from "yume-dsl-rich-text";

parseIncremental(
    source: string,
    options?: IncrementalParseOptions,
): IncrementalDocument

createIncrementalSession(
    source: string,
    options?: IncrementalParseOptions,
    sessionOptions?: IncrementalSessionOptions,
): {
    getDocument: () => IncrementalDocument;
    applyEdit: (
        edit: IncrementalEdit,
        newSource: string,
        options?: IncrementalParseOptions,
    ) => IncrementalSessionApplyResult;
    applyEditWithDiff: (
        edit: IncrementalEdit,
        newSource: string,
        options?: IncrementalParseOptions,
        diffOptions?: IncrementalDiffRefinementOptions,
    ) => IncrementalSessionApplyWithDiffResult;
    rebuild: (newSource: string, options?: IncrementalParseOptions) => IncrementalDocument;
}

interface IncrementalSessionApplyResult {
    doc: IncrementalDocument;
    mode: "incremental" | "full-fallback";
    fallbackReason?:
        | "INVALID_EDIT_RANGE"
        | "NEW_SOURCE_LENGTH_MISMATCH"
        | "EDIT_TEXT_MISMATCH"
        | "UNKNOWN"
        | "INTERNAL_FULL_REBUILD"
        | "FULL_ONLY_STRATEGY"
        | "AUTO_COOLDOWN"
        | "AUTO_LARGE_EDIT";
}

interface IncrementalSessionApplyWithDiffResult extends IncrementalSessionApplyResult {
    diff: TokenDiffResult;
}

interface TokenDiffResult {
    isNoop: boolean;
    patches: Array<{
        kind: "insert" | "remove" | "replace";
        oldRange: { start: number; end: number };
        newRange: { start: number; end: number };
    }>;
    unchangedRanges: Array<{
        oldRange: { start: number; end: number };
        newRange: { start: number; end: number };
    }>;
    ops: Array<
        | {
        kind: "splice";
        path: Array<{ field: "root" | "children" | "args"; index: number }>;
        field: "root" | "children" | "args";
        oldRange: { start: number; end: number };
        newRange: { start: number; end: number };
        oldNodes: StructuralNode[];
        newNodes: StructuralNode[];
    }
        | {
        kind: "set-text" | "set-escape" | "set-raw-content";
        path: Array<{ field: "root" | "children" | "args"; index: number }>;
        oldValue: string;
        newValue: string;
    }
        | {
        kind: "set-implicit-inline-shorthand";
        path: Array<{ field: "root" | "children" | "args"; index: number }>;
        oldValue?: boolean;
        newValue?: boolean;
    }
    >;
    dirtySpanOld: { startOffset: number; endOffset: number };
    dirtySpanNew: { startOffset: number; endOffset: number };
}

diff fields:

• isNoop: true when this edit produced no structural token changes (patches and ops are both empty).
• patches: top-level token-index patches (possibly multiple segments).
• unchangedRanges: top-level unchanged islands.
• ops: path-aware structural operations for nested children / args edits and scalar updates.
• dirtySpanOld / dirtySpanNew: source-coordinate dirty spans in old/new documents.

If precise diff refinement cannot be completed internally, applyEditWithDiff(...) conservatively falls back to a whole-tree replacement diff instead of leaving the session in a partially updated state.

Additionally, for very large documents that already took the full-fallback path, applyEditWithDiff(...) may choose the conservative whole-tree diff route directly to avoid pathological deep-refinement cost.

Strategy is set via the third parameter sessionOptions of createIncrementalSession:

// always full rebuild (debugging / baseline comparison)
const session = createIncrementalSession(source, {handlers}, {
    strategy: "full-only",
});

// always attempt incremental first, fall back only on failure
const session = createIncrementalSession(source, {handlers}, {
    strategy: "incremental-only",
});

All sessionOptions fields are optional — unspecified fields use their defaults. To tweak individual auto-strategy parameters, pass only the ones you want to override:

// auto strategy (default), but raise the large-edit threshold to 40%
const session = createIncrementalSession(source, {handlers}, {
    maxEditRatioForIncremental: 0.4,
});

// auto strategy, shorter cooldown window
const session = createIncrementalSession(source, {handlers}, {
    fullPreferenceCooldownEdits: 6,
});

// override multiple at once
const session = createIncrementalSession(source, {handlers}, {
    maxEditRatioForIncremental: 0.4,
    fullPreferenceCooldownEdits: 6,
    softZoneNodeCap: 128,
    diff: {
        refinementDepthCap: 64,
        maxComparedNodes: 20000,
        maxAnchorCandidates: 128,
        maxOps: 512,
        maxSubtreeNodes: 256,
        maxMilliseconds: 8,
    },
});

// one-call override for a hot path (does not mutate session defaults)
const withDiff = session.applyEditWithDiff(
    edit,
    newSource,
    undefined,
    {maxMilliseconds: 4, maxOps: 256},
);

• refinementDepthCap controls how deep applyEditWithDiff(...) continues nested path-aware refinement.
• Higher values preserve more fine-grained nested ops on deep trees, but cost more on pathological depth.
• Lower values switch earlier to coarse splice ops, usually improving worst-case latency.
• Invalid values are normalized to the default.
• maxComparedNodes / maxAnchorCandidates / maxOps / maxSubtreeNodes / maxMilliseconds together define the default hard diff budget: the goal is “be fine-grained in normal cases, degrade immediately once cost escapes”, not “force a minimal patch for every input”.
• Internally these budgets are accumulated and propagated through an explicit DiffBudgetState; budget exhaustion is treated as an expected degradation condition, not as a throw/catch exception path, and the diff directly falls back to coarse splice / conservative output.
• When the edit actually stays on the incremental update path, applyEditWithDiff(...) first reuses the incremental dirty-zone source window and only refines diff inside that root window; outside the dirty window it treats the root ranges as stable instead of refining the whole root tree.

Validation and consistency failures:

• INVALID_EDIT_RANGE: offsets are invalid against old doc.source (negative, inverted, or out of bounds).
• NEW_SOURCE_LENGTH_MISMATCH: newSource.length doesn't match applying the edit delta.
• EDIT_TEXT_MISMATCH: edit.newText doesn't match newSource at startOffset.
• UNKNOWN: unexpected error (bugs or external exceptions).

Strategy-driven fallbacks:

• AUTO_LARGE_EDIT: the auto strategy detected a too-large edit ratio.
• AUTO_COOLDOWN: the auto strategy is temporarily in full-preferred mode.
• FULL_ONLY_STRATEGY: you asked for strategy: "full-only".
• INTERNAL_FULL_REBUILD: the guarded incremental path escalated to a full rebuild internally.

The top half is the session-layer strategy gate; the bottom half is what happens inside the internal incremental updater.

flowchart TD
    A["applyEdit(edit, newSource)"] --> B{strategy?}
    B -->|full - only| C["full rebuild"]
    C --> C1["✗ full-fallback · FULL_ONLY_STRATEGY"]
    B -->|auto / incremental - only| D{"auto &&<br/>editRatio > threshold?"}
    D -->|yes| E["full rebuild"]
    E --> E1["✗ full-fallback · AUTO_LARGE_EDIT"]
    D -->|no| F{"auto &&<br/>in cooldown window?"}
    F -->|yes| G["full rebuild"]
    G --> G1["✗ full-fallback · AUTO_COOLDOWN"]
    F -->|no| H["enter internal incremental path ↓"]
    H --> V{"edit range<br/>valid?"}
    V -->|no| V1["✗ full-fallback · validation error code"]
    V -->|yes| FP{"options fingerprint<br/>compatible?"}
    FP -->|no| FP1["✗ full-fallback · INTERNAL_FULL_REBUILD"]
    FP -->|yes| ZE{"zone list<br/>non-empty?"}
    ZE -->|no| ZE1["✗ full-fallback · INTERNAL_FULL_REBUILD"]
    ZE -->|yes| ZL{"zone count<br/>> 1?"}
    ZL -->|no| ZL1["✗ full-fallback · INTERNAL_FULL_REBUILD<br/>(low-zone guard)"]
    ZL -->|yes| TG{"tail coverage<br/>safe?"}
    TG -->|no| TG1["✗ full-fallback · INTERNAL_FULL_REBUILD"]
    TG -->|yes| DZ["locate dirty zone range"]
    DZ --> RP["reparse dirty window<br/>+ right-boundary stabilization"]
    RP --> BG{"expansion bytes<br/>over budget?"}
    BG -->|yes| BG1["✗ full-fallback · INTERNAL_FULL_REBUILD"]
    BG -->|no| RZ{"right-side zones<br/>to reuse?"}
    RZ -->|no| AS
    RZ -->|yes| SP{"right-seam<br/>probe passes?"}
    SP -->|no| SP1["✗ full-fallback · INTERNAL_FULL_REBUILD"]
    SP -->|yes| AS["assemble: left as-is +<br/>reparsed result +<br/>right lazy delta shift"]
    AS --> OK["✓ incremental"]

session.applyEdit(...) returns a safe result object — no try/catch needed.

const result = session.applyEdit(edit, newSource);
if (result.mode === "full-fallback") {
    console.log(result.fallbackReason);
}

The API processes one edit per call. If you have a batch of edits:

1. Apply them sequentially via session.applyEdit(...), in the same order you used to build the final newSource. (Recommended — easier to debug.)
2. Merge the patches into a single large edit in the editor layer, then make one call.

If your UI stores diff payloads for inspection, avoid frame-level overwrite coalescing. In a single frame with multiple edits, a later isNoop: true can overwrite a meaningful earlier diff.

A common editor pipeline — "structural cache picks the region, token-walker tokenizes it locally":

1. Keep doc.tree / doc.zones updated with session.applyEdit(...).
2. Use a structural query (e.g. nodeAtOffset / enclosingNode) to select a region.
3. Call parseSlice(fullText, node.position, dsl, tracker) to re-parse only that region into TextToken[].

This works because doc.tree positions always point into the full newSource (trackPositions is forced on internally).

Since 1.2.4, right-side zone reuse uses lazy delta shifting: each zone stores only an O(1) offset delta, and node positions are materialized on first consumer access to doc.tree or doc.zones[i].nodes. Consecutive head-of-file edits automatically stack deltas without intermediate materialization.

• Head-of-file edits are no longer expensive. Previously every edit deep-copied the entire right-side subtree; now it just records a number.
• Middle/tail edits are as fast as before (the dirty range is small by definition).
• The only "payment" happens when you actually traverse doc.tree for rendering/export.

Incremental performance depends on zone count — more zones mean a smaller dirty window per edit.

Problem: Before 1.2.4, buildZones(...) only split at raw / block nodes. Pure-inline documents produced just 1 zone, equivalent to a full rebuild.

Solution: 1.2.4 introduced softZoneNodeCap (default 64) in the internal incremental zone builder — consecutive non-breaker nodes exceeding this count are automatically split. Split points always land on node boundaries, preserving the seam probe invariant. The public buildZones(...) API is unaffected.

const session = createIncrementalSession(source, {handlers}, {
    softZoneNodeCap: 128,  // larger zones → less signature overhead, but wider dirty windows
});

When the previous snapshot has ≤ 1 zone, the incremental path skips directly to a full rebuild — with only 1 zone there's nothing to reuse, and all incremental overhead is wasted. Returns INTERNAL_FULL_REBUILD.

Same machine and methodology as the Performance page. Edit position fixed at head of file (worst case), single character replacement.

Zones are not "more is always better" — each zone carries constant overhead for signature computation, assembly stitching, and delta shifting. The sweet spot is a few hundred to a few thousand; above ~10K the per-zone overhead starts to dominate.

Rule of thumb: once full rebuild latency becomes noticeable (say > 10 ms), zone splitting brings incremental into usable range.

flowchart TD
    N["top-level node sequence"] --> IS{"node is<br/>raw / block?"}
    IS -->|yes| FH["flush accumulated soft zone"]
    FH --> HB["hard boundary: own zone"]
    HB --> IS
    IS -->|no| ACC["accumulate into current soft zone"]
    ACC --> CAP{"accumulated count<br/>≥ softZoneNodeCap?"}
    CAP -->|yes| FS["flush current soft zone"]
    FS --> IS
    CAP -->|no| IS

• Large document + high edit frequency: use a session, let lazy shifting accumulate, and only read doc.tree on render frames.
• Large document + only need local tokens: pair with parseSlice(...) for the best ROI.
• Treat options as immutable. parseIncremental(...) snapshots options into doc.parseOptions (handlers' plain object/array fields are cloned recursively).
• Keep your handlers reference stable across edits — same function references mean the same fingerprint, which means no unnecessary full rebuilds.

• oldEndOffset is exclusive. Off-by-one errors here silently trigger a fallback.
• Offsets are JavaScript string offsets (UTF-16 code units). If you're bridging from a native editor that uses byte offsets or Unicode scalar values, you'll need to convert.
• edit.newText must match newSource at startOffset, otherwise you get EDIT_TEXT_MISMATCH. This is the most common mistake — usually it means newSource was built with a stale version of the document.
• Raw/block forms depend on real newlines. "\\n" is not the same as "\n" — this trips people up when constructing edits programmatically.

For normal editor integration, prefer createIncrementalSession(...) directly.

const doc = parseIncremental(source, options);

doc guarantees:

• doc.source is exactly the input source.
• doc.tree is structural parse output with trackPositions forced on.
• doc.zones is built from doc.tree via buildZones(...).
• Every node used by zones carries source positions (offset/line/column) relative to doc.source.

How options behaves:

• trackPositions is ignored/overridden to true on this path.
• options are captured into an internal snapshot (handlers plain object/array fields are cloned recursively).
• Reuse compatibility is checked by internal option fingerprinting on effective parser behavior (syntax/forms/handler function identities).
• Re-passing a new options object with behavior-equivalent values won't force a full rebuild.

The update is zone-bounded, not an arbitrary window reparse. Understanding these rules helps explain why certain cases fall back to full rebuild.

1. Initial dirty range (zone-level):
   • findDirtyRange locates the dirty window in a single linear pass that simultaneously detects overlap and records the insertion index (fused into one traversal since 1.2.5).
   • If the edit overlaps existing zones: start with "overlap range + one zone left + one zone right". The extra padding accounts for edits that change how adjacent zones parse.
   • If the edit is a pure insertion with no overlap: start from the insertion neighbors with a one-zone left lookbehind.
2. Right-boundary stabilization:
   • After reparsing the dirty window, if the reparsed end offset doesn't match the current dirty-window end offset, expand right by one zone and try again. Repeat until stable or EOF.
3. Right-seam probe for reuse:
   • Before stitching right-side zones back in, re-parse a small window at the stitch boundary and compare the produced zone signatures against the old ones.
   • Signatures use bounded sampling hash (first/last 32 characters), making per-node comparison O(1).
   • If the probe mismatches, fall back to a full rebuild.
4. Expansion budget guard:
   • The stabilization loop tracks cumulative reparsed bytes (2 × newSource.length budget). Once exceeded, bail out to full rebuild.
5. Lazy delta shifting (1.2.4+):
   • Right-side zones that pass the seam probe store a pendingDelta instead of deep-copying. Node positions are materialized on first consumer access. Consecutive edits stack deltas automatically.
6. Malformed-snapshot fallback:
   • If the input snapshot looks unsafe (empty zones, broken tail coverage), fall back to full rebuild immediately.
7. Full-rebuild path constant optimization (1.2.5+):
   • cloneParseOptions deferred until incremental path is confirmed.
   • The already-built positionTracker is reused during fallback rebuild.

Incremental API version differences, upgrade impact, and recovery-shape changes such as the 1.4.1 EOF unclosed-inline behavior now live on:

• Version Semantics Notes

export interface IncrementalEdit {
    startOffset: number;
    oldEndOffset: number; // exclusive, in old source offsets
    newText: string;
}

startOffset / oldEndOffset are offsets into the old text (session.getDocument().source). newText must match newSource.slice(startOffset, startOffset + newText.length).

export interface IncrementalDocument {
    source: string;
    zones: readonly Zone[];
    tree: readonly StructuralNode[];
    parseOptions?: IncrementalParseOptions;
}

export type IncrementalParseOptions = Omit<
    StructuralParseOptions,
    "trackPositions" | "baseOffset" | "tracker"
>;

It is nearly identical to StructuralParseOptions, minus three fields managed internally by the incremental engine. Available fields:

In practice, if you already have an options object for parseStructural, you can pass the same object to createIncrementalSession — the omitted fields are ignored internally (TypeScript will flag them at the type level, but semantically they have no effect).

export interface IncrementalSessionOptions {
    strategy?: "auto" | "incremental-only" | "full-only";
    sampleWindowSize?: number;
    minSamplesForAdaptation?: number;
    maxFallbackRate?: number;
    switchToFullMultiplier?: number;
    fullPreferenceCooldownEdits?: number;
    maxEditRatioForIncremental?: number;
    softZoneNodeCap?: number;
    diff?: IncrementalDiffRefinementOptions;
}

export interface IncrementalDiffRefinementOptions {
    refinementDepthCap?: number;
    maxComparedNodes?: number;
    maxAnchorCandidates?: number;
    maxOps?: number;
    maxSubtreeNodes?: number;
    maxMilliseconds?: number;
}

softZoneNodeCap controls the internal zone builder's soft splitting threshold. Default is 64. Minimum effective value is 2 (clamped internally).

The public export surface for incremental types is intentionally small. Diff fragment types are not exported as standalone root-level names — the stable contract is TokenDiffResult. Derive finer types from fields:

type DiffPatch = TokenDiffResult["patches"][number];
type DiffRange = TokenDiffResult["unchangedRanges"][number];
type DiffOp = TokenDiffResult["ops"][number];

import {
    createIncrementalSession,
    parseIncremental,
} from "yume-dsl-rich-text";

import type {
    IncrementalDiffRefinementOptions,
    IncrementalDocument,
    IncrementalEdit,
    IncrementalParseOptions,
    IncrementalSessionApplyResult,
    IncrementalSessionApplyWithDiffResult,
    IncrementalSessionOptions,
    TokenDiffResult,
} from "yume-dsl-rich-text";