frame_assembly_and_compositor_spec - mark-ik/graphshell GitHub Wiki

Frame Assembly and Compositor β€” Interaction Spec

Date: 2026-02-28 Status: Canonical interaction contract Priority: Active (Stages 1–4 complete; Stage 4b implemented)

Related:

  • ASPECT_RENDER.md
  • ../../../archive_docs/checkpoint_2026-03-22/graphshell_docs/implementation_strategy/aspect_render/2026-02-20_embedder_decomposition_plan.md
  • ../PLANNING_REGISTER.md Β§0, Β§0.10
  • ../workbench/workbench_layout_policy_spec.md Β§3.2, Β§3.4
  • ../viewer/viewer_presentation_and_fallback_spec.md
  • ../../TERMINOLOGY.md β€” CompositorAdapter, TileRenderMode, Composition Pass, Surface Composition Contract

Adopted standards (see 2026-03-04_standards_alignment_report.md Β§Β§3.6, 3.7)):

  • OpenTelemetry Semantic Conventions β€” diagnostics channels (GL state violation, chaos mode, frame timing) follow OTel naming and severity conventions
  • OSGi R8 β€” TileRenderMode resolution and CompositorAdapter capability dispatch follow OSGi capability vocabulary

1. Scope

This spec defines the canonical contracts for:

  1. Composition pass ordering β€” the three-pass frame model and ownership rules.
  2. TileRenderMode β€” the render pipeline classification for each tile.
  3. CompositorAdapter β€” lifecycle, GL state isolation, callback ordering.
  4. Embedder decomposition seams β€” EmbedderCore vs RunningAppState boundary.
  5. Frame loop coordination β€” begin/layout/paint ownership.

This spec covers the current render architecture and what is being actively built. The egui_glow β†’ egui_wgpu UI-backend cut has already landed; the remaining renderer work is the deeper WebRender/runtime bridge path rather than the old egui backend swap.

Historical note: the earlier composited-viewer contract note has been archived. Any still-relevant future-work ideas from its Appendix A are tracked in ../PLANNING_REGISTER.md Β§0.10 rather than in a separate active render-contract doc.


2. Composition Pass Model Contract

Every node viewer pane tile frame is composed in three ordered passes:

Pass Name Owner Content
1 UI Chrome Pass Render aspect Tab bar, badge overlay, focus/hover rings, tile chrome
2 Content Pass Viewer (via CompositorAdapter) Web content, native content, or fallback surface
3 Overlay Affordance Pass Render aspect Selection rings, diagnostic affordances, pointer hit targets

Invariant: Pass ordering is Graphshell-owned sequencing and must not rely on incidental egui layer behavior. The three-pass order is always Chrome β†’ Content β†’ Overlay. No pass may mutate the output of a preceding pass.

Invariant: The CompositedTexture TileRenderMode renders Overlay Affordance Pass content over the composited texture in the pipeline; the NativeOverlay mode renders affordances in tile chrome/gutter regions because native content owns its own window region.


3. TileRenderMode Contract

TileRenderMode is the runtime-authoritative render pipeline classification for a node viewer pane tile. It is resolved from ViewerRegistry at viewer attachment time.

TileRenderMode =
  | CompositedTexture   -- Servo GL texture composited into egui frame
  | NativeOverlay       -- native window overlay (e.g. Wry); owns its own region
  | EmbeddedEgui        -- egui-native content (native viewers, metadata card)
  | Placeholder         -- no viewer attached; fallback surface

Invariant: Every NodePaneState must have a TileRenderMode set at viewer attachment time. A tile must never enter the compositor dispatch without a resolved TileRenderMode.

Resolution path: ViewerRegistry::resolve_mode(viewer_id) -> TileRenderMode. The resolved value is stored on NodePaneState and updated only when the viewer changes.

3.1 Per-Mode Compositor Behavior

Mode Content Pass behavior Overlay Pass behavior
CompositedTexture Invoke CompositorAdapter GL callback; render Servo texture into tile rect Render affordances over texture in tile rect
NativeOverlay No GL callback; native content owns its region Render affordances in chrome/gutter only (not over native content)
EmbeddedEgui Render via normal egui widget tree Render affordances as egui overlays
Placeholder Render fallback surface (loading indicator, error state, empty state) Render affordances over fallback

3.2 Navigation Geometry Contract Note

Workbench layout policy may derive visible navigation geometry that differs from the logical navigation-region remainder when overlay-form Navigator hosts occlude part of the workbench surface.

Current runtime rule:

  1. Runtime consumers that make visibility or placement decisions should honor the full visible navigation rect set rather than collapsing immediately to one fallback rect.
  2. This includes viewport culling, diagnostics geometry summaries, and floating overlay placement.
  3. Graph/input consumers follow the same rule via the runtime-carried typed visible-region contract.
  4. The remaining follow-on work is promotion of visible navigation geometry into a first-class pane/render contract so the canonical render model itself no longer speaks in single-rect terms where a derived visible region set is the real authority.

See workbench_layout_policy_spec.md Β§3.4 for the authoritative definition of logical navigation region versus visible navigation geometry.


4. CompositorAdapter Contract

CompositorAdapter wraps backend-specific content callbacks. It owns:

  • Callback ordering within the Content Pass.
  • GL state isolation (save/restore GL state around callbacks; the callback must not leak GL state into the egui render path).
  • Clipping and viewport contracts (the callback renders only within the tile rect).
  • The post-content overlay hook (called after content callback, before the Overlay Affordance Pass).

4.1 GL State Isolation Invariant

Every CompositorAdapter callback must:

  1. Save GL state before invoking the content callback.
  2. Restore GL state after the callback returns, regardless of whether the callback succeeded or panicked.
  3. The egui render path must observe consistent GL state before and after a compositor callback.

GL state diagnostics: Violations (leaked GL state) must be observable via the diagnostics channel schema. A GL_STATE_MISMATCH channel event is emitted when state restoration fails.

4.2 Callback Registration

Content callbacks are registered at viewer attachment time. A callback is a function of type:

fn render_content(tile_rect: Rect, clip_rect: Rect, gl_state: &mut GlStateGuard)

Callbacks are unregistered at viewer detachment time. A tile with no registered callback falls back to Placeholder rendering.


5. Embedder Decomposition Seam Contract

The historical RunningAppState monolith conflated embedder and app-layer responsibilities. The decomposition boundary is:

EmbedderCore (embedder responsibility) RunningAppState (app-layer responsibility)
Servo instance EmbedderCore (owned)
Windows map (HashMap<EmbedderWindowId, EmbedderWindow>) AppPreferences
Event loop waker Gui handle
WebViewDelegate + ServoDelegate trait impls Intent queues
WebDriver channels Gamepad provider (app-level routing)

EmbedderCore invariant: EmbedderCore must not hold references to graphshell app state (preferences, intent queues, graph data). It communicates via GraphSemanticEvent emissions only.

GraphSemanticEvent is the clean boundary: All semantic information crossing from the embedder layer into the graphshell graph/app layer must pass through GraphSemanticEvent. No direct embedder→graph calls.

5.1 Decomposition Stage Summary

Stage Status Description
1 Complete Semantic bridge extraction (semantic_event_pipeline.rs)
2 Complete Toolbar decomposition (7 focused submodules)
3 Complete CompositorAdapter extraction (wraps rendering paths EmbedderCore exposes)
4a βœ… Complete shell/desktop/ui/gui.rs frame orchestration isolated from workbench layout driving
4b βœ… Complete EmbedderCore/RunningAppState boundary closure plus host-runtime service extraction (WebDriverRuntime, GamepadRuntime, EmbedderWindow internal service splits)

Historical Stage 4 sequencing note: Compositor pass-order correctness and GL-state diagnostics hardening landed before the Stage 4b decomposition follow-through. See viewer/2026-02-26_composited_viewer_pass_contract.md Appendix A for the sequencing rationale.


6. Frame Loop Coordination Contract

The egui frame loop coordinates three phases:

  1. Begin frame β€” ctx.begin_frame(input): owned by the Render aspect entry point.
  2. Layout pass β€” widget tree construction and layout computation: owned by Workbench (tile tree traversal) + Viewer (per-tile content).
  3. Paint β€” ctx.end_frame() + GPU surface present: owned by the Render aspect.

Invariant: Workbench must not call ctx.begin_frame or ctx.end_frame. It participates in the layout pass only. Frame start/end are Render aspect responsibilities.

Invariant: The compositor dispatch (CompositorAdapter callbacks) runs within the layout pass, after the tile rect is known and before end_frame. It does not run between begin/end frame boundaries.


7. Acceptance Criteria

Criterion Verification
Pass ordering is always Chrome β†’ Content β†’ Overlay Test: instrument pass entry points β†’ verify ordering across 100 frames
GL state is identical before and after a compositor callback Test: capture GL state before/after callback β†’ no differences
GL state mismatch emits diagnostics channel event Test: inject a leaking callback β†’ GL_STATE_MISMATCH event in diagnostics
Every NodePaneState has a resolved TileRenderMode Test: attach viewer β†’ tile_render_mode field is non-null
NativeOverlay affordances render in chrome/gutter only Test: NativeOverlay mode β†’ no affordance draw calls inside native content rect
EmbedderCore emits no direct graph mutations Architecture invariant: no graph_app.* calls from EmbedderCore module
GraphSemanticEvent is the only crossing point Architecture invariant: all embedder→app communication passes through GraphSemanticEvent
Compositor callback is unregistered on viewer detach Test: detach viewer β†’ callback list is empty; tile falls back to Placeholder

8. C4 β€” Render Pass Sharing: Design

The following section records the design analysis for eliminating redundant render pass open/close cycles per frame (C4 in the compositor optimization sequence).

Context

C3 batched command encoding β€” all draw calls now share a single CommandEncoder per frame via pending_encoder. But each draw_instanced() call still creates its own begin_render_pass() / drop cycle. A typical frame has 50-200 draw calls across ~10-20 unique render targets, meaning 50-200 render pass cycles where ~10-20 would suffice. Render pass creation is expensive: the GPU driver must resolve load/store ops, flush tile memory (on tiled GPUs), and validate attachments. Sharing one render pass per target eliminates this overhead.

Lifetime Analysis: Why a Guard Struct Doesn't Work

RenderPass<'a> borrows the CommandEncoder mutably. If we stored both the encoder and the pass on WgpuDevice, we'd have a self-referential borrow. Rust forbids this.

A scoped closure approach also fails: while the render pass borrows the encoder, we also need to call methods on wgpu_dev (get_pipeline, create_bind_groups, etc.) β€” and those require &mut self. If the encoder is still on wgpu_dev, we can't take &mut self.

A draw-command-recording approach (collect Vec<DrawCmd> per target, replay in one pass) avoids lifetimes but adds per-frame allocation and borrow complexity from TextureBindings<'_>.

Final Design: Take/Put-Back Encoder

The take/put-back encoder is the most pragmatic solution:

  1. Encoder is an Option<CommandEncoder> field on WgpuDevice.
  2. Renderer calls wgpu_dev.take_encoder() β€” takes encoder out, making wgpu_dev freely usable for &self and &mut self methods.
  3. Renderer creates render pass locally from the taken encoder, issues N draws.
  4. Render pass is dropped (closing the pass on the GPU).
  5. Renderer calls wgpu_dev.return_encoder(encoder) β€” puts encoder back.

The key insight: while the encoder is taken out, wgpu_dev is fully accessible for pipeline lookups, buffer creation, bind group creation. The render pass is a local variable in the renderer β€” clean lifetime.

// renderer code:
let mut encoder = wgpu_dev.take_encoder();
let target_uniforms = wgpu_dev.create_target_uniforms(target_w, target_h);
{
    let mut pass = encoder.begin_render_pass(&desc);
    for batch in batches {
        let pipeline = wgpu_dev.ensure_pipeline(variant, blend, depth, fmt);
        let (bg0, bg1) = wgpu_dev.create_bind_groups(..., &target_uniforms);
        let ibuf = wgpu_dev.create_instance_buffer(data);
        pass.set_pipeline(pipeline);
        pass.set_bind_group(0, &bg0, &[]);
        pass.set_bind_group(1, &bg1, &[]);
        pass.set_vertex_buffer(0, wgpu_dev.unit_quad_vb.slice(..));
        pass.set_vertex_buffer(1, ibuf.slice(..));
        pass.set_index_buffer(wgpu_dev.unit_quad_ib.slice(..), Uint16);
        pass.draw_indexed(0..6, 0, 0..count);
    }
} // pass dropped here
wgpu_dev.return_encoder(encoder);

New WgpuDevice Methods

impl WgpuDevice {
    pub fn take_encoder(&mut self) -> wgpu::CommandEncoder { ... }
    pub fn return_encoder(&mut self, encoder: wgpu::CommandEncoder) { ... }
    pub fn create_target_uniforms(&self, width: u32, height: u32)
        -> (wgpu::Buffer, wgpu::Buffer) { ... }
    pub fn ensure_pipeline(&mut self, variant: WgpuShaderVariant,
        blend_mode: WgpuBlendMode, depth_state: WgpuDepthState,
        target_format: wgpu::TextureFormat) -> Option<&wgpu::RenderPipeline> { ... }
    pub fn record_draw<'a>(&mut self, pass: &mut wgpu::RenderPass<'a>, ...) { ... }
}

Renderer Changes

  • draw_passes_wgpu() picture cache loop: take encoder β†’ create_target_uniforms β†’ create render pass β†’ loop batches (prepare bind groups + instance buffer + record draw) β†’ drop pass β†’ return encoder.
  • draw_cache_target_tasks_wgpu(), draw_clip_batch_list_wgpu(), draw_quad_batches_wgpu(): same take/put-back pattern per target.
  • Texture cache targets: the per-target loop in draw_passes_wgpu creates ONE pass and passes it to each helper function.
  • Scissor reset: when switching from a scissored draw to non-scissored within the same pass, explicitly set_scissor_rect(0, 0, w, h) to reset.

Implementation Order

  1. Add take_encoder(), return_encoder(), create_target_uniforms(), ensure_pipeline() to WgpuDevice. Keep existing draw_instanced() working.
  2. Migrate picture cache rendering in draw_passes_wgpu().
  3. Migrate texture cache target rendering (one pass per target across all helpers).
  4. Optionally unify composite path (render_composite_instances_to_view()).
  5. Remove old draw_instanced() if all callers migrated.

Risk Notes

  • Medium: Render pass must be created with correct depth attachment upfront β€” has_opaque is computed before the batch loop, so this info is available.
  • Medium: Scissor rect state persists within a pass; must explicitly reset.
  • Low: Pipeline/bind group changes within a pass are standard wgpu operations.
⚠️ **GitHub.com Fallback** ⚠️