Build and Verification - limztudio/NWBLot GitHub Wiki
Build projects and targets from the repository root with python -m launcher (python3 on Linux). The launcher selects the matching preset, configures when needed, and builds the requested targets and their dependencies. Users do not need to run CMake commands themselves.
Real native launches require an executable artifact in the selected CMake build and configuration's File API reply, unless --executable supplies an explicit path. Missing metadata or artifacts fail before launch; dry runs retain executable naming previews. The pipeline likewise accepts explicit tool paths or --tool-directory.
The build uses CMake with Ninja Multi-Config and LLVM/Clang internally. CMakePresets.json defines the supported build variants, and each target's CMakeLists.txt defines its sources and link dependencies.
- Windows x64
- Windows ARM64
- Linux x64
All supported targets are 64-bit. The project configurations are dbg, opt, and fin.
Common tools:
- Git with Git LFS
- CMake 3.25 or newer
- Ninja
- LLVM/Clang, including
clang,clang++,llvm-ar, andllvm-ranlib, with C++23-or-newer standard-librarystd::expectedsupport -
lld-linkfor Windows - Python 3.8 or newer for
python -m launcherand the string-literal compiler pipeline - The compiler host's LLVM C API shared library matching both C and C++ Clang exactly
-
slangcwhenNWB_BUILD_PIPELINE=ON, which is the default - A Vulkan loader and compatible Vulkan driver for runtime rendering
Configuration selects a supported latest/C++23 frontend flag and compiles a real std::expected success/error probe. Both compiler mode and target standard library must provide __cpp_lib_expected >= 202202L; accepting a flag alone does not qualify a toolchain. global/expected.h enforces the same requirement, and global/compile.h explicitly rejects non-Clang compilers. CMake owns real compiler/frontend detection; headers query Clang attributes/builtins and use current SIMD implementations directly, without compiler-ID/frontend adapter macros or older-compiler emulations. The C++23 filesystem Path API always supports char8_t input, and genericU8String() returns std::u8string.
Run git lfs pull after checkout so the LFS-managed third-party runtime files are present.
The Windows and Linux toolchains find Ninja on PATH, through NWB_NINJA, or below NWB_NINJA_ROOT. The Windows toolchain can also use the Ninja installation supplied by Visual Studio.
LLVM tools are found on PATH, through NWB_LLVM_ROOT or LLVM_ROOT, and on Windows through supported Visual Studio or standalone LLVM installation paths.
The launcher resolves CMake in this order:
- Its
--cmakeargument. -
CMAKE_COMMAND. -
__cmake/tool-venv/when that local environment exists. -
cmakeonPATH.
The asset pipeline resolves slangc through NWB_SLANGC_EXECUTABLE, __cmake/slang, VULKAN_SDK, or PATH. Missing slangc is fatal when NWB_BUILD_PIPELINE=ON.
Windows hosts require Windows 10 version 1709 or newer. The launcher requires IsWow64Process2 for native architecture discovery; frame creation uses current Per-Monitor v2 DPI APIs, including SetProcessDpiAwarenessContext.
Install Visual Studio 2022 Build Tools or Visual Studio 2022 with:
- The Desktop development with C++ workload.
- The x86/x64 MSVC tools for x64 targets.
- The MSVC ARM64 tools for ARM64 targets.
- A Windows SDK that provides headers, libraries,
rc.exe, andmt.exe.
CMake, Ninja, and LLVM may be installed separately or supplied through Visual Studio. The presets invoke clang and clang++, use lld-link, and target the Windows/MSVC ABI, so the Microsoft C++ runtime and Windows SDK remain required. First-party compilation requires a Clang-based compiler; GNU-style Clang/AppleClang and clang-cl frontend handling remain current. Native cl.exe compilation is unsupported.
The Vulkan SDK is optional because Vulkan headers and Volk are vendored. Installing it is useful for slangc, validation layers, and Vulkan diagnostic tools.
In addition to the common tools, install development packages for:
- X11
- zlib
- libcurl
- oneTBB, including
tbbandtbbmalloc
X11 is required even when the desktop session uses Wayland. The Wayland backend is enabled when wayland-client, xkbcommon, wayland-scanner, and the xdg-shell protocol file are available. Missing Wayland dependencies disable that backend without disabling X11. Pass -D NWB_ENABLE_WAYLAND=OFF to a launcher build command to skip Wayland discovery explicitly.
The launcher selects these presets from the host platform, --arch, --domain, and --config. The matrix documents the underlying build trees and the matching CTest presets; users can choose normal builds through launcher options.
| Platform and domain | Configure preset | Build preset pattern | CTest preset pattern |
|---|---|---|---|
| Windows x64 full | windows-clang-x64 |
windows-clang-<config> |
windows-clang-<config> |
| Windows x64 engine | windows-clang-engine-x64 |
windows-clang-engine-<config> |
— |
| Windows x64 Testbed | windows-clang-testbed-x64 |
windows-clang-testbed-<config> |
— |
| Windows ARM64 full | windows-clang-arm64 |
windows-clang-arm64-<config> |
windows-clang-arm64-<config> |
| Windows ARM64 engine | windows-clang-engine-arm64 |
windows-clang-engine-arm64-<config> |
— |
| Windows ARM64 Testbed | windows-clang-testbed-arm64 |
windows-clang-testbed-arm64-<config> |
— |
| Linux x64 full | linux-clang-x64 |
linux-clang-<config> |
linux-clang-<config> |
| Linux x64 engine | linux-clang-engine-x64 |
linux-clang-engine-<config> |
— |
| Linux x64 Testbed | linux-clang-testbed-x64 |
linux-clang-testbed-<config> |
— |
Use dbg, opt, or fin in place of <config>. Full presets enable both the Testbed and internal tests. Engine presets omit both. Testbed presets build the application without the internal test tree. Separate *-namesym-* configure and build presets support name-symbol capture.
The build commands configure automatically and stop after building. CTest remains the separate test execution tool; the launcher has no CTest subcommand.
python -m launcher build all --arch arm64 --config dbg
ctest --preset windows-clang-arm64-dbgpython -m launcher build all --arch x64 --config dbg
ctest --preset windows-clang-dbgpython3 -m launcher build all --arch x64 --config dbg
ctest --preset linux-clang-dbgThe maintained Linux baseline uses the full linux-clang-x64 configure preset and lowercase multi-config labels. Use the explicit CTest build tree command when recording a verification result:
python3 -m launcher build all --arch x64 --config dbg
ctest --test-dir __cmake/build/linux-clang-x64 -C dbg --output-on-failureThe launcher finds CMake through the discovery rules above. If CTest is not on PATH, use __cmake/tool-venv/bin/ctest when that local environment exists. The explicit CTest command is equivalent to the debug test preset but records the configured build tree and configuration explicitly.
Cook convergence is a separate check. After a dbg full build, run the pipeline twice with independent output and cache roots over both asset roots. Prefer --skip-build with the already-built tool directory so the comparison does not start a nested CMake build:
python3 -m launcher pipeline --skip-build \
--tool-directory __exec/linux/x64/full/dbg \
--asset-root impl/assets \
--asset-root CoolStuff/Testbed/assets \
--output-directory /tmp/nwblot-cook-a/out \
--cache-directory /tmp/nwblot-cook-a/cache \
--configuration dbgRepeat the command with independent nwblot-cook-b output and cache roots, then compare the two published output directories. Record the result in two stages: each pipeline invocation must first succeed; only then is a byte-level difference (for example, from diff -qr) a determinism failure. A failed cook and two successful but byte-different cooks are distinct outcomes.
The pipeline's --asset-type defaults to graphics when omitted. Explicitly empty and unsupported values are rejected, including malformed internal build requests.
Build focused targets through the launcher. Use a native project shortcut with --build-only, or the generic build command for executable, library, and aggregate targets:
python -m launcher testbed --build-only --arch arm64 --config dbg
python -m launcher build nwb_pipeline --arch arm64 --domain engine --config dbg
python -m launcher build nwb_global_tests nwb_cpu_task_tests nwb_gpu_task_tests --arch arm64 --config dbg| Configuration | Configuration macro | Clang optimization | Debug information | Frame pointer |
|---|---|---|---|---|
dbg |
NWB_DEBUG |
-O0 |
Enabled | Kept |
opt |
NWB_OPTIMIZE |
-O2 |
Enabled | Kept |
fin |
NWB_FINAL |
-O3 |
Enabled | Omitted |
CMake emits these canonical configuration macros directly, without intermediate aliases. First-party headers do not infer NWB modes from DEBUG, _DEBUG, NDEBUG, or _NDEBUG. Required compiler/runtime uses of those spellings remain toolchain contracts.
Configuration names are exactly lowercase dbg, opt, and fin. In a single-config tree, omitted or explicitly empty CMAKE_BUILD_TYPE defaults to dbg. Every other nonempty value fails configuration, including Debug, Release, unknown names, OFF, 0, and case variants; a false-like CMake value is still explicit input. Multi-config trees expose the same canonical names. Imported third-party libraries retain mappings to their vendor Debug, Release, and related artifact configurations. Those mappings are current dependency integration and do not create alternate first-party configuration names.
Clang's ordinary C/C++ literal pooling applies on every target; clang-cl also receives /GF. NWB_OBFUSCATE_STRING_LITERALS=ON is the default in all three configurations. The compiler launcher emits optimized LLVM bitcode, pools and encodes eligible surviving literal globals, and emits the requested object with its original code-generation level and without another LLVM optimization pipeline. One runtime object per executable or shared library decodes its records before application static initialization. Literal-backed constexpr StringView retains frontend compile-time evaluation and direct pointer/length access, with no per-access decode guard or heap string allocation.
Enabled builds require a Ninja or Makefiles generator, Python 3.8 or newer, and a loadable compiler-host LLVM C API library with exactly the same major, minor, and patch version as both Clang compilers. Configuration checks the library's required API exports and rejects missing libraries, version mismatches, unsupported generators, and competing compiler/linker launchers. Library discovery searches beside the compiler and in toolchain library directories; supply an explicit path if needed:
python -m launcher build all --config opt -D "NWB_LLVM_C_LIBRARY=C:/Program Files/LLVM/bin/LLVM-C.dll"Use -D NWB_OBFUSCATE_STRING_LITERALS=OFF for ordinary pooling without encoding. Both settings are cached; pass -D NWB_OBFUSCATE_STRING_LITERALS=ON to reenable the default. Unsupported transformed object modes, including LTO and multi-source object commands, fail explicitly, and failures preserve a prior successful object.
Equal eligible narrow, UTF-16, and UTF-32 records pool by bytes, character width, and alignment within each linked image. Encoding adds headers, alignment, writable storage, and one startup decode pass, so net memory savings depend on duplicate literal volume. Named arrays, address-significant or explicitly retained objects, optimized immediates, debug data, and name-symbol data are outside the transformation's coverage. Decoded text remains readable in memory. COFF, ELF, and Mach-O handling in the compiler pipeline does not add application-platform support beyond the target list above.
The focused nwb_string_literal_compiler_tests CTest exercises early static-constructor reads, escaping cross-translation-unit views, UTF-16/UTF-32 embedded NUL content, named-array identity, recursive response files, rejected modes/version mismatches, and failure recovery:
python -m launcher build nwb_global_tests --arch arm64 --config opt
ctest --preset windows-clang-arm64-opt -R "^nwb_string_literal_compiler_tests$"The main options default to:
NWB_BUILD_PIPELINE=ONNWB_BUILD_UTILITIES=ON-
NWB_BUILD_LOGSERVER=ONon Windows and Linux -
NWB_BUILD_LOADER=ONon Windows and Linux -
NWB_BUILD_TESTS=OFFunless a full or explicit test configuration enables it
Configure trees are stored at:
__cmake/build/<configure-preset>/
Runtime artifact roots are:
engine: __exec/<platform>/<arch>/<config>/
full: __exec/<platform>/<arch>/full/<config>/
testbed: __exec/<platform>/<arch>/testbed/<config>/
namesym: __exec/<platform>/<arch>/namesym/<config>/
Internal asset caches use __build_obj/c/<platform>/<arch>/<domain>/. Test runtimes, captures, and other CTest artifacts live below the matching configure tree's Testing/ directory.
The root launcher/ package is the common entry point for builds and runnable workflows. build <targets...> configures when required and builds one or more targets. Native project and utility shortcuts support --build-only to stop after building; without that flag, they resolve the executable artifact and launch it from the correct working directory. On Windows the launcher selects the native host architecture by default; use --arch x64 or --arch arm64 for an explicit target.
python -m launcher build all --config opt
python -m launcher testbed --build-only --config dbg
python -m launcher testbed --config dbg
python -m launcher run testbed --config opt
python -m launcher pipeline --help
python -m launcher pipeline --config dbg --asset-root impl/assets CoolStuff/Testbed/assets --output-directory runtime/res
python -m launcher smoke --profiles
python -m launcher smoke gi-test --backend native
python -m launcher profilesEach runnable project or utility owns its leaf launch.py; category launchers dispatch to those files, and the root launcher discovers their commands. The repository launcher itself is a split Python package (launcher/): immutable data models in models.py, repo discovery/routing in discovery.py, platform probing in host.py, CMake settings in cmake_settings.py, build orchestration in build.py, process lifecycle in process.py, profile sessions in profile.py, command dispatch in dispatch.py, CLI construction in cli.py, and shared literals in constants.py, re-exported through the launcher/__init__.py package root. CoolStuff/Testbed/launch.py owns testbed, utilities/fbx_to_nwb/launch.py owns fbx-to-nwb, utilities/tex_conv/launch.py owns tex-conv, utilities/font_builder/launch.py owns font-builder, and utilities/ui_skin/launch.py owns ui-skin. The UI skin generator is a Python authoring workflow, so use python -m launcher ui-skin --help for its generator options. Specialized pipeline, smoke, and A/B commands keep their own options rather than inheriting native --build-only behavior.
Common build controls are:
-
--config dbg,opt, orfinselects the configuration. -
--domain full,engine, ortestbedselects the build variant; the default isfull. -
--jobs 8sets parallel build work. -
--configure alwaysrefreshes configuration;--configure neverrequires an existing configured tree. -
-D KEY=VALUEsupplies a configuration setting and triggers configuration. -
--dry-runprints the plan without configuring, building, or starting applications.
A fresh custom build directory is configured automatically from the selected preset. Relative --build-dir paths resolve against the selected repository root (--repo-root may select another checkout):
python -m launcher build testbed --config opt --build-dir "__cmake/build/custom project"Build-only commands reject application arguments after -- and contradictory --skip-build requests before making changes. Installed toolchain dependencies are still required. See the source-managed launcher guide for the complete build workflow.
The root launcher discovers pipeline/launch.py as a normal runnable directory. Build and asset options share one argument list. The pipeline launcher configures when needed with NWB_BUILD_PIPELINE=ON, builds nwb_pipeline, and runs dependency_computer, asset_builder, then asset_gatherer. Use --skip-build with --tool-directory to reuse already-built tools; --dry-run prints planned commands without writing artifacts. The three C++ binaries are dependency_computer, asset_builder, and asset_gatherer.
Surfel-GI A/B captures use python -m launcher renderer-baseline surfel-gi-complex (closed 20-body skylight scene, NWB_GI_SMOKE_COMPLEX_SCENE=1, freeze frame 360; extend settling with NWB_GI_SMOKE_MIN_SETTLE_SECONDS, e.g. 100, keeping the override and GPU-validation choice identical between reference and candidate runs; see tests/ab/renderer_baseline/README.md).
Use --with-profile on a native project run to build and start nwb_logserver with the application. Combined with --build-only, it only prepares the profiling dependencies and starts neither the application nor the profiling server. Use --run-seconds <N> for a bounded capture. python -m launcher profiles is the authoritative list of interactive and A/B workflows; hardware-specific A/B commands keep their reports under .cozter/out/ab-results/.
CTest presets belong to full configure presets. A focused test run uses -R:
python -m launcher build nwb_global_tests nwb_cpu_task_tests nwb_gpu_task_tests --arch arm64 --config dbg
ctest --preset windows-clang-arm64-dbg -R "^nwb_global_tests$"
ctest --preset windows-clang-arm64-dbg -R "^nwb_cpu_task_tests$"
ctest --preset windows-clang-arm64-dbg -R "^nwb_gpu_task_tests$"The test tree is organized as:
-
tests/unit/: deterministic subsystem and policy tests. -
tests/integration/: cross-module, asset, pipeline CLI, process, and tool tests. -
tests/smoke/: selected graphics runtime, capture, and hardware-dependent workflows. Thetests/smoke/ui_layer/widget-capture fixture separates arithmetic data lookup, native scene construction, posted native input, log decoding, and pixel acceptance;combo_probe_common.pyowns the shared accepted-snapshot and displayed-pixel contract (ComboProbe, marker encoding, and tolerance) borrowed by both the plain and searchable combo probes. -
tests/ab/: manually launched performance and behavior comparisons.
Unit tests cover concrete boundaries and regressions: invalid or unsupported input, exact bounds, overflow, empty or degenerate data, aliasing, stale generations, ownership, cancellation, reentry, concurrency, ordering, and failure recovery. This scope also applies to Python parser tests and harness self-tests registered as unit suites. Remove routine defaults, getters/setters, argument forwarding, name mappings, successful round trips, and source-text wiring checks. Keep successful setup and controls when they prove a boundary or recovery, and remove unused fixtures and source registrations with deleted cases. Actual CLI and runtime workflows remain in integration/smoke coverage; standalone benchmark executables remain explicit performance workloads under tests/performance/ and tests/smoke/.
Keep test sources and fixtures in their owning domain. CPU and GPU task suites live under tests/unit/task/cpu (nwb_cpu_task_tests) and tests/unit/task/gpu (nwb_gpu_task_tests), selected public graphics contracts under tests/unit/graphics/, and renderer ownership under tests/unit/ecs_graphics/. The unit suites exercise their remaining runtime boundaries; source searches alone do not qualify as boundary coverage. Keep executable and suite names stable where possible, and name cases after the failure scenario they exercise.
Window-capture and hardware-dependent tests need a usable display and compatible Vulkan device. Tests that cannot exercise their required environment return the configured skip code instead of reporting hardware coverage. Capture files are written below __cmake/build/<configure-preset>/Testing/smoke/<config>/.
The renderer requires VK_EXT_descriptor_buffer. Selected graphics smoke coverage may skip on devices that cannot satisfy the renderer's required feature set. Use dbg or opt for GPU validation and inspect the generated logs in addition to the process exit status. CSG, caustic, and UI capture CTests select --gpu-validation in dbg/opt and --no-gpu-validation in fin. Their wrappers default validation on for manual runs; the generic window-capture wrapper defaults it off. Final executables reject --gpudbg. The dedicated Testbed and refraction gpudbg capture tests are registered only for dbg. Verify requested validation through Loader: GPU debug validation enabled in the captured log. Continue rejecting warnings, errors, assertions, and native validation diagnostics; external loader/layer environment configuration remains part of runtime admission.
Reflection and refraction coverage runs as capture smokes against nwb_reflection_smoke and nwb_refraction_smoke: nwb_reflection_capture_smoke plus the rough, temporal, optical, feedback, and feedback-diagnostics-off variants, and nwb_refraction_capture_smoke / nwb_refraction_gpudbg_capture_smoke. Suites select route, budget, and filter cases, validate accepted-route diagnostics and framebuffer content, and skip on adapters without the required hardware ray queries. See tests/smoke/REFLECTION.md and docs/refraction.md for the feature and capture contracts. Temporal-omission coverage uses nwb_reflection_temporal_omission_smoke with nwb_reflection_temporal_omission_analysis_unit; the reusable frozen A/B timing workflow is renderer_ab_benchmark.py with analysis tests in tests/integration/renderer_ab_benchmark_tests.py.
The transparent CSG capture fixture uses an unlit green center receiver with reflection and refraction disabled. It derives oracle regions for its early, middle, and late poses and verifies receiver identity, so lighting and optical background transport cannot obscure the clipping and coverage checks. Renderer unit coverage in nwb_ecs_graphics_tests checks executable ownership, boundary, and failure-recovery behavior through current public contracts.
Stress timing (stress_timing_smoke.py and stress_timing_smoke_tests.py) runs a twenty-body workload with ten glass and ten opaque bodies at 1280x900. It uses a five-second warmup and thirty-second accepted-presentation count in fixed view (yaw 0.6, fixed simulation delta) or rotating mode (--animate, wall-clock spinning). CPU diagnostics use the --cpu-diagnostics mode of stress_timing_smoke.py (CPU+GPU timing capture via smoke_cpu_gpu_timing.py, diagnostic only) with analysis in stress_timing_smoke_tests.py. stress_timing_smoke.py --minimum-fps 60 enforces strictly greater count/time FPS and retains the run evidence when the requested threshold is missed. Record the selected quality settings, route, binaries, assets, and host with each result; an average FPS threshold is not a per-frame deadline guarantee. Disabled-policy runs require Loader: hardware ray tracing disabled before device creation, RendererSystem: dispatched software shadow traversal, and feature-specific software-route evidence. See tests/smoke/software_raytracing.md for route admission and tests/smoke/caustic_quality_smoke.py / tests/integration/caustic_quality_smoke_tests.py for caustic quality checks.
Building testbed automatically builds nwb_pipeline and cooks assets from:
impl/assetsCoolStuff/Testbed/assets
Cooked files are published to the selected runtime root's res directory. Runtime graphics loads cooked assets; it does not compile authoring shaders directly.
The optional nwb_namesym target creates readable sidecars for optimized and final diagnostics. Build it from a normal full build tree:
python -m launcher build nwb_namesym --arch arm64 --config optUse --arch x64 on Windows x64, or python3 -m launcher build nwb_namesym --arch x64 --config opt on Linux. The default full domain selects the normal build tree. Python 3 is required. GUI capture workloads may skip without a display, while the headless pipeline cook capture remains available. Expected sidecars include dependency_computer.namesym, asset_builder.namesym, asset_gatherer.namesym, and testbed.namesym. Before new capture workloads run, old sidecars must be removed successfully; a cleanup failure aborts capture and publication, preserving the previously published destination sidecars.
Register each new C or C++ file in the owning target's nearest CMakeLists.txt with target_sources, then refresh configuration and build it through python -m launcher build <target> --configure always with the appropriate architecture, domain, and configuration options. Visual Studio can open the repository root directly as a CMake project, but it is not a second source-list authority: do not add or maintain duplicate Visual Studio project source rules.
Before changing project code, read .helper/standard.md and the other applicable .helper guides. Use project wrappers from global/ at engine boundaries; BitCast from global/bit.h is the wrapper for std::bit_cast.
Packages are vendored as flat top-level directories under 3rd_parties/ and are wired through repository CMake without configure-time downloads. Every package records its version, source, update time, and integration notes in nwb_update.txt. Follow Third-Party Packages when changing a vendored dependency.
Buffer byte-range behavior is covered by ordinary enabled unit tests in tests/unit/task/gpu/ (no DISABLED_* cases), including task_graph_buffer_range_tests.cpp (independent intervals, partial-overwrite prefix/tail collection, same-task overlap seeding, terminal-interval exports, cross-queue fan-in on intersecting bytes, and late ownership/terminal-export grouping) and the partial-DAG placement cases in task_graph_queue_partial_dag_tests.cpp (multiword score preservation across queue moves and rebuilds, sparse-row scratch bounds, relation-state reset on rebuild, assignment exclusion without local assignment, and mask-allocation avoidance for ordered/unused reachability). Those tests carry correctness assertions and no wall-clock thresholds; they exercise the CPU pipeline, while native GPU frame performance requires a representative rendering workload. Run them with an optimized build:
python -m launcher run nwb_gpu_task_tests --config opt --working-directory . -- --gtest_output=xml:gpu_task_tests.xmlThe launcher builds and runs the target; the test executable writes the XML report in the repository root. See core/task/gpu/README.md for the buffer byte-range contract and docs/gpu_task_performance_history.md for preserved historical measurement evidence.