Diagnostics and Invariants - limztudio/NWBLot GitHub Wiki
This page captures engine-wide rules that matter when changing NWBLot. For operational logging, crash packaging, GPU crash reports, telemetry, and frame graph capture, see Diagnostics and Telemetry.
Logging is centralized through the logger client/server system, with standalone logger support for tools and non-server runs.
Tools and non-interactive executables should route output through logger APIs instead of ad hoc console output when they are part of the engine/tooling ecosystem.
Name-symbol decoding is a logger-server responsibility. Build-mode executables may write .namesym sidecars, and the server loads them to decode hash tokens in formatted log lines. The logger client must not depend on the core/common/name_symbols.h implementation or carry client-side name-symbol upload bookkeeping in normal operation.
Global Expected<T, E>, Unexpected<E> and MakeUnexpected(error) from global/expected.h define fallible producer results. The default error is empty Failure; meaningful domain enums, ErrorCode, diagnostic aggregates and guaranteed borrowed error text remain explicit. Return answers, owned candidates and coherent reports directly, and check admission before value/error access. Successful false, zero, empty and documented no-work values remain valid. Preserve supplied arenas, borrowed lifetimes, publication/partial-progress semantics, complete noexcept contracts and terminal exception unwinding. Genuine buffer reuse, lifecycle mutation and required SDK/callback signatures remain in/out contracts; fallible mutators can return Expected<void, E>. First-party builds require actual C++23-or-newer std::expected library support. See the source result guide.
The common application entry point owns the process-wide exception boundary. Production tools use InvokeTerminalEntry from global/terminal_entry.h so CLI parse errors and unexpected exceptions stay at that boundary. Expected producer failures use typed Expected errors; genuine mutators and callbacks retain their explicit failure contracts. Unexpected exceptions unwind to that entry boundary; intermediate code must not catch, log or convert one into an ordinary failure and then continue execution. Native worker exceptions terminate at their thread boundary; the helper never transports them.
First-party production C++ catch clauses belong only in global/terminal_entry.h. A source policy rejects local recovery, logging-and-continue, rollback-before-rethrow, and deferred worker-exception delivery. Cleanup should be RAII and failure-atomic. Irreversible publication must complete before a potentially throwing observer, and unwind cleanup must avoid callbacks that could replace the original exception.
Scheduler owners join outstanding CPU work during normal teardown. They use callback-free drain() while another exception is already unwinding. Native worker exceptions terminate at their thread; they are not stored for later observation at wait().
The loader installs crash capture early, before CLI parsing or logging setup. Crash packages can include:
- process crash data
- metadata such as runtime and GPU debug mode
- GPU crash reports
- vendor GPU dumps when available
- minimal
gpu_crash.txtreports when richer diagnostics are unavailable
GPU diagnostics integrate with vendor paths where present:
- NVIDIA Aftermath dumps
- AMD Radeon GPU Detective dumps
- Vulkan device fault details where supported
The crash path should still emit a useful text report when richer vendor diagnostics are unavailable.
The frame owns telemetry capture and upload plumbing. Renderer systems can contribute frame graph information through IFrameGraphContributor.
Telemetry and frame graph capture belong to the frame/runtime layer, while renderer subsystems contribute structured data about their own work.
Several current engine rules matter when changing code:
- Runtime graphics consumes cooked shader bytecode; shader source compilation belongs only to cook targets.
- Shader-language compilation stays in cook targets, including
impl/assets_shader,NWB_COOK, and theNWB_BUILD_PIPELINEtools. - Runtime asset bytes go through
IFilesystem. Do not restoreVolumeSessionor mount cooked volumes by scanning source directories. - Shared CPU/GPU shader ABI constants live under
impl/assets/graphicsin macro-only headers usable by C++ and Slang. - Material parameters are schema-driven by
.bindfiles and typed generated accessors. - Material and shader parameter binding is typed-only at binding
6. - Shaders that include a generated material bind require
NWB_MATERIAL_TYPED_BINDING == 1. - Generated typed material accessors at binding
6are the renderer's material-binding interface. - ECS infrastructure stays generic; production scene/content components live in
impl. - Renderer camera state comes from scene camera components, not input/controller components.
-
GraphicsRuntimeowns a required backend for its lifetime; required local device access is treated as an invariant. -
Frameowns oneCpuTaskSchedulerand oneGpuTaskScheduler. Subsystems borrow those services; do not restoreThreadPool,JobSystem, or private worker pools. - Vulkan feature reporting must reflect enabled feature structs and loaded function pointers, not only extension names.
- CPU/software Vulkan devices are rejected for windowed runtime initialization.
- Asset and shader authored text files use CRLF line endings in the repository.
- Configure and build through
python -m launcher, then run the matching CTest preset with the platform prerequisites listed in Build and Verification. - Project code should request shutdown through
ProjectRuntimeContext::requestQuit. - Runtime graphics should not parse authored
.nwb,.bind,.slang,.slangi,.surface, or.bxdffiles. - Engine-owned renderer shader infrastructure and shared CPU/GPU ABI assets belong under
impl/assets. - Material interfaces, surfaces, and BXDF policy belong to the project asset root that authors them.
- Project-specific authored assets belong under project trees such as
CoolStuff/Testbed/assetsortests/smoke/assets. - The logger server is the sole normal name-symbol resolver; do not move sidecar loading or symbol-table upload ownership back into the logger client.
First-party C and C++ code must not conceal a function-call result merely to silence an unused-result warning. Do not suppress direct or locally stored results through static_cast<void>(...), C-style void casts, [[maybe_unused]], or std::ignore.
Handle the result according to the operation's contract: propagate it, branch on it, log/report the failure, or otherwise make the outcome observable.
NWBLot-owned utility, configuration, diagnostic, namespace, and shader feature macro APIs use NWB_; application macro APIs use their own prefix, such as TESTBED_. Preserve external/compiler spellings and existing g_* shader resource-view aliases. CMake emits NWB_DEBUG, NWB_OPTIMIZE, and NWB_FINAL directly for dbg, opt, and fin; project-owned C++ abstract interfaces retain the lowercase interface declaration annotation. global/compile.h supplies the portable struct expansion only when an external SDK has not already defined it. The annotation is an intentional exception to the macro-prefix rule and remains the canonical spelling during compatibility cleanup. See Architecture for engine/application namespace ownership.
Borrow text through StringView or the matching BasicStringView<CharT> when a known length fits the API. Static literals can use constexpr views. Choose borrowed or owned member storage from the backing lifetime and CPU/memory cost, preserve external C-string termination requirements, and never retain a view into temporary or asynchronously released bytes. Mark a C++ function noexcept only after its complete implementation, callees, callbacks, and construction/destruction establish that guarantee.
First-party code uses project wrappers from global/ for standard-library facilities. Representation-preserving casts use BitCast<To>(value) from global/bit.h; direct std::bit_cast belongs only inside that wrapper.
CpuTaskScheduler does not capture, store, or defer worker exceptions. Unexpected native-worker failures terminate at the native thread. Inline or caller-thread failures unwind to the application-entry boundary. The owning boundary still joins outside of a task whose completion is required by the wait.
drain() is noexcept teardown: it stops admission, skips queued callbacks, and waits for outstanding work without observers. A task cannot wait on itself, an ancestor, dependent work that needs its completion, or a scope containing any ancestor. Use cancel() plus wait() for ordinary scope cancellation; scope drain() aborts the shared service because application unwind is terminal. See CPU Task Scheduler.
Material metadata fields are defined by impl/assets_material/metadata.h, including transparent, two_sided, and refractive.
Material metadata requires a cook-resolved bxdf. A material declares exactly one of surface or explicit shaders: the surface path generates the material pixel shader and assigns the shared engine mesh shader, while shaders remains the explicit stage map.
refractive is a caster classification, not a replacement for transparent. The material .surface hook returns refractionIor, shadowAbsorptionTint, and renderCoverage. A transparent surface with finite refractionIor above 1.0 (nwbRefractionIorEligible in impl/assets/graphics/refraction/ior.slangi) becomes a camera-refraction primary: the renderer retains at most one primary per pixel, omits it from AVBOIT occupancy/extinction/color accumulation, and resolves its transmission separately. renderCoverage is the AVBOIT opacity parameter, not an optical transmission mask, so clear glass may report zero coverage.
Reflection and presentation settings are external control boundaries: ValidateReflectionSettings and ValidatePresentationSettings reject invalid values before they reach shaders, history state, or the display transform.
Material-bind parsing normalizes typed block declarations into structs and instances, clears parser output state between parses, enforces unique symbols, and requires string-only bind-attribute arguments.
Authored .bind files do not need a material_bind asset; declaration.
Half typed-layout support is append-only. Half fields are packed as 16-bit values, and typed blocks are padded to 4 bytes.
The material resource slice permits texture2d and sampler only in material_constant blocks. These fields declare no .bind attributes; their block-scoped asset.parameters values must be engine/ or project/ paths to typed Texture or Sampler assets. Cooked resource references carry kind/source, block/field identity, asset-name hash, and a constant-byte offset; they never serialize a device-lifetime descriptor handle. The renderer loads and caches the typed asset, resolves its global-heap slot, and patches a working copy of the constant bytes. Resource fields are not MaterialInstanceComponent overrides.
Renderer shading remains material-driven. AVBOIT transparency stays consistent with opaque helper logic, including close-up aliasing mitigation, per-material occupancy/extinction/accumulate shading for surface-authored transparent materials, and transparent multi-object smoke regression coverage.
Transparent shadow transmittance is material-driven. The hardware opaque fast pass handles opaque visibility. Staged hardware transparent evaluation and software GPU-BVH traversal use the cook-generated surface dispatch to obtain optical values and integrate RGB transmittance. Both feed the shared shadow visibility contract. The software path also supplies the hardware-frame fallback, and no-RT light-space shading resolves its supported depth-captured crossings through the same material surface contract.
Refractive caustics are material-driven. Hardware RT and software GPU-BVH caustic producers should converge through the same accumulator and resolve target, and skinned refractors should use live repacked normals for trace work.
Renderer CSG work must keep preview and commit viability aligned across sequential cuts with deterministic CPU-side rebuild or remesh and attribute-aware wall/cap handling.
Deferred lighting always samples shadow visibility. If neither hardware RT nor software BVH writes the visibility image in a frame, the renderer must clear it to all-lit white.
Deferred lighting always samples caustic irradiance. If no caustic producer writes it in a frame, the renderer must clear it to black so additive lighting is a no-op.
Renderer pipeline caches must account for framebuffer/render-target compatibility, not just material or shader identity.
Cross-command-list resource state may move only through a valid CommandListResourceStateHandoff. Export state only at producer close, after keepInitialState restore barriers. The handoff carries final state plus sharing and exclusive-owner metadata, but it is not synchronization or a resource-lifetime mechanism: an accepted QueueSubmissionToken or ordered same-queue submission must establish producer-before-consumer execution, and referenced resources must remain alive until the consumer opens.
Concurrent Graphics/AsyncCompute resources must never receive ownership transfers. An exclusive cross-family resource requires paired release/acquire barriers and the matching accepted token; fan-in must reject incompatible final state, sharing contract, or owner instead of inferring ownership from queue order. If accepted Compute work releases shadowVisibility, selected software causticIrradiance, or surfelIrradiance but the intended Graphics consumer is rejected, recovery must return every released output to Compute before reuse. Failure to accept that recovery suspends rendering until resource or device recreation rather than assuming an owner.
Software shadow, caustic, and surfel-GI traversal share the float NwbBvhNode ABI.
Software shadow traversal uses front-to-back child ranking, near-child inline descent, far-child stack deferral, contiguous triangle-index Load3 fetches, and root-only scene/mesh slab tests. Parent child-ranking tests provide the per-node bounds decision.
Software scene-BVH construction uses a binned-SAH split policy with primitive-count leaf weights and a median split for degenerate centroid ranges.
Runtime graphics consumes cooked SPIR-V bytecode. The neutral nwb_shader_bytecode target owns core/graphics/spirv_entry_point.h and .cpp and declares a public nwb_common dependency, alongside the standard internal target defaults. Shader asset runtime/cook targets use it for admission, and nwb_graphics_backend uses the same parser for module validation. Payload encoding, concrete types and codecs remain in impl/assets_shader; provider-specific headers remain behind graphics selection. Project-specific authored assets, including material interfaces, surfaces, and BXDFs, live under project trees such as CoolStuff/Testbed/assets and tests/smoke/assets. Engine-owned renderer shader infrastructure and shared CPU/GPU ABI assets live under impl/assets.
Shader metadata declares a concrete physical-stage type, such as compute_shader or pixel_shader; generic shader declarations and authored asset.stage are rejected. Runtime interface IShader shares entry-point/bytecode storage among final concrete types with static constexpr s_Stage. Concrete admission validates the current payload, complete bounded SPIR-V stream, and exact entry point for that physical stage before publishing state. Wrong-stage or malformed loads preserve previously loaded storage. Read unaligned byte views without a temporary word copy; retain backend shader-module validation through the selected graphics API and keep validation out of steady rendering paths. Do not report malformed modules as ordinary missing-entry-point failures.
Shader archive roles remain distinct from physical types: mesh_compute selects ComputeShader, while mesh_object_vertex selects VertexShader. Known stage references and loaders use concrete types; heterogeneous stage arrays use AssetRef<IShader> and explicit dynamic-stage loading. Share payload/codec/compiler machinery without adding a per-object stage field, duplicate bytecode, allocation layer, or per-frame virtual stage dispatch.
Shader dependency checks use stable alias roots and record BOM presence from the bytes already read. Source checksums include a once-per-append fingerprint of the configured compiler executable/version and shared argument contracts. Each shader cache record is one checksum-named .spv, with no checksum sidecar; cache hits and fresh output require complete bounded SPIR-V validation for the exact entry point and physical stage. Each compiler invocation has its own temporary directory, requires readable diagnostics, cleans BOM overlays on failure as well as success, and atomically promotes validated output. Authored define names are ASCII identifiers, and values must round-trip the variant grammar without semicolons, NUL/CR/LF, or surrounding whitespace. Cooked asset payloads are expected to be deterministic for isolated cook checks.
Shader dependency discovery splices continued lines and masks comments before resolving literal includes. Reject unplanned macro includes, #include_next, and #import. Any explicitly caller-declared generated macro include requires that caller to prepare and track its source, as with the current CSG evaluator.
Shader volume writing calls the single direct Slang compiler module for fingerprinting and compilation; the separate ShaderSourceDependencies module owns source scanning and include resolution. ShaderCook validates already-parsed documents and owns variants/dependencies/checksums. Do not reintroduce unused compiler interfaces, factories, injection paths, compile forwarding methods, or production file parser convenience overloads. Test-owned parsing helpers remain under tests/. Preserve all 14 current physical stages, including those without authored fixtures.
Texture metadata contains semantic format/dimension/color/alpha choices and its same-directory UASTC .tex filename. The cooker derives the complete contiguous mip chain, fixed block geometry, slice counts, offsets, sizes, and HDR alpha range with checked arithmetic, then validates the exact sidecar byte count before publication. Authored layout, mip records/counts, copied HDR flags, and version/revision fields are rejected. Runtime graphics loads the cooked typed Texture payload rather than the source pair.
CSG shape type ids are a canonical name-derived cooked/runtime ABI. Do not restore insertion-order ids; both the cooker and runtime registry must reject a canonical-id collision.
Cook success and byte-level determinism should be tracked separately.
The graphics volume cook uses a pack manifest. Registry-backed typed assets are serialized into content-addressed .nwbobj files under the cook cache, and each manifest entry carries the payload size, payload hash, and cook-key hash that the volume writer verifies before staging the bytes. asset_builder publishes standalone .nwba artifacts plus assets.list; asset_gatherer packages those artifacts through IFilesystem and staged volume promotion. Runtime loading must continue to consume the published volume, not the cook cache or .nwba directory.
Whole-file reads size the opened stream, avoiding a separate successful-path stat that can describe a different file after replacement. Unrepresentable write lengths fail before destination truncation; writers close explicitly so buffered failures affect the returned result. Redirected process execution prepares allocating argument text before acquiring native handles and checks wait completion plus the exit-code query before returning a child result.
Metascript parser views must retain valid backing storage through declaration-vector growth. The first declared asset name borrows the lexer source for synchronous parsing rather than a movable declaration string.
All test sources, fixtures, assets, fixture selectors, and test-only helpers belong under tests/. Production material schemas and shaders contain no fixture-only resource path; test materials use ordinary typed asset references and the normal cook/load/bind contract. Unit cases cover concrete edges and regressions; routine defaults, forwarding, and successful round trips do not need unit cases. Graphics tests exercise existing public contracts after normal backend selection, without provider-specific tests, private provider fixtures, a test backend, or production test hooks.
Tests in this repository use direct GTest TEST bodies. Prerequisite checks that gate later logic should use ASSERT_*, string equality should use EXPECT_STREQ, and direct pointer/null assertions are preferred unless a compound predicate is intentional.
Negative cook tests that expect validation failures should be gated to NWB_FINAL when non-final builds may trigger debug breaks.
Window capture smoke tests may skip in headless Linux environments without a usable display server.
Smoke diagnostic environment overrides should be read through tests/smoke/smoke_environment.h helpers. Individual smoke projects and probes should not grow new fixed-size local environment buffers for float/text parsing.