Feature composition - thekaveh/NNx GitHub Wiki
26. Feature-composition matrix
Generated by scripts/check_feature_composition.py --render from
docs/feature-composition.yaml (the expectation) and a run's evidence report.
Which feature combinations NNx verifies, refuses or leaves unverified, per device / dtype / dependency profile. A cell is verified only when every listed test passed its setup, call and teardown on the published profile at these exact sources. An import, a skip or an xfail never counts. Without a current report every cell reads unverified.
Evidence (cpu): 25ed8c56fe140488 — sources 5163cbae8556 (git 4f39c77647a2), profile linux / Python 3.12 / torch 2.13.0 / cpu.
Training controls
| ID | Features | Profile | Expected | Status | Evidence |
|---|---|---|---|---|---|
| TC-01 | gradient accumulation + trailing partial window | cpu / fp32 / core | Steps only at window ends; a trailing partial window steps once at the epoch boundary. | verified — every listed test passed (cpu evidence) | tests/test_pass2_f_series.py::test_f2_accumulate_grad_batches_steps_trailing_partial_cycle |
| TC-02 | update cap + gradient accumulation + budgeted search | cpu / fp32 / optuna | An update cap of 2 with accumulate_grad_batches=3 allows no third committed step; a partial window is no update. | verified — every listed test passed (cpu evidence) | tests/test_search.py::test_an_update_cap_binds_and_a_partial_window_is_no_update |
| TC-03 | early stopping + stateful resume | cpu / fp32 / core | A resumed run's restored patience stops training at the epoch the uninterrupted run stopped at. | verified — every listed test passed (cpu evidence) | tests/test_component_resume.py::test_restored_patience_stops_at_the_uninterrupted_epoch |
| TC-04 | DDP + gradient accumulation + uneven ignore_index + validation | cpu / fp32 / torch.distributed gloo | Two ranks equal one process on the union batch (updates and globally weighted records). | verified — every listed test passed (cpu evidence) | tests/test_ddp_adapter.py::test_two_ranks_match_the_global_batch_reference |
| TC-05 | finalize_step paradigm steps + gradient accumulation | cpu / fp32 / core | Refused with a ValueError (accumulation is not handled by the imperative paradigm steps). | unsupported — the negative test passes (cpu evidence) | tests/test_step_helpers.py::test_finalize_step_raises_on_gradient_accumulation |
| TC-06 | FP16 automatic mixed precision + CUDA | cuda / fp16 / core | Trains through a GradScaler on CUDA, with finite losses and scaler updates. | unverified — profile not run: needs cuda | tests/test_cuda_evidence.py::test_fp16_amp_trains_through_a_grad_scaler_on_cuda |
| TC-07 | DDP + torch.compile | cpu / fp32 / torch.distributed gloo | Refused on every rank before any run starts. | unsupported — the negative test passes (cpu evidence) | tests/test_ddp_adapter.py::test_preflight_refusals_fail_on_every_rank_before_any_run |
| TC-08 | DDP + HistoryJournal | cpu / fp32 / torch.distributed gloo | Refused on every rank before any run starts; DDP keeps the eager history. | unsupported — the negative test passes (cpu evidence) | tests/test_ddp_adapter.py::test_preflight_refusals_fail_on_every_rank_before_any_run |
| TC-09 | DDP + budgeted search + experiment plans | cpu / fp32 / torch.distributed gloo | Refused on every rank of a multi-rank group before any study, factory call or run; a one-rank group behaves as no group. | unsupported — the negative test passes (cpu evidence) | tests/test_ddp_adapter.py::test_single_process_entry_points_refuse_a_multi_rank_grouptests/test_ddp_adapter.py::test_single_process_entry_points_run_in_a_one_rank_group |
| TC-10 | BF16 automatic mixed precision + CUDA | cuda / bf16 / core | Trains on a bf16-capable CUDA device without a GradScaler, with finite losses. | unverified — profile not run: needs cuda | tests/test_cuda_evidence.py::test_bf16_trains_without_a_scaler_on_cuda |
| TC-11 | DDP + NCCL + two GPUs | cuda / fp32 / nccl | Two GPUs over NCCL equal the single-process reference within FP32 device tolerance. | unverified — profile not run: needs cuda | tests/test_cuda_evidence.py::test_two_gpu_nccl_parity_matches_the_single_process_reference |
| EX-01 | synthetic classification example + end to end | cpu / fp32 / core | The example runs to completion as a subprocess. | verified — every listed test passed (cpu evidence) | tests/test_examples_smoke.py::test_representative_examples_run_end_to_end[01_synthetic_classification.py]tests/test_examples_smoke.py::test_bounded_example_helpers_execute[01_synthetic_classification.py-native_nll_workflow] |
| EX-05 | custom train step + autoencoder example | cpu / fp32 / core | The custom-step example runs to completion as a subprocess. | verified — every listed test passed (cpu evidence) | tests/test_examples_smoke.py::test_representative_examples_run_end_to_end[05_custom_train_step_autoencoder.py]tests/test_train_step_hook.py::test_autoencoder_style_step_trains_end_to_end |
| EX-26 | custom eval step + non-finite metric | cpu / fp32 / core | The custom-eval example runs, and its non-finite-metric helper asserts the finite-metric fallback. | verified — every listed test passed (cpu evidence) | tests/test_examples_smoke.py::test_representative_examples_run_end_to_end[26_custom_eval_step.py]tests/test_examples_smoke.py::test_bounded_example_helpers_execute[26_custom_eval_step.py-nonfinite_metric_workflow] |
Checkpoint and resume
| ID | Features | Profile | Expected | Status | Evidence |
|---|---|---|---|---|---|
| CR-01 | warm resume + list loader | cpu / fp32 / core | A warm resume equals the uninterrupted run. | verified — every listed test passed (cpu evidence) | tests/test_train_integration.py::test_warm_resume_list_loader_matches_uninterrupted |
| CR-02 | torch.compile + resume | cpu / fp32 / core | A resume builds a fresh compiled wrapper around the reloaded module. | verified — every listed test passed (cpu evidence) | tests/test_compile_execution.py::test_resume_rebuilds_the_wrapper |
| CR-03 | DDP + resume + rank RNG | cpu / fp32 / torch.distributed gloo | Two epochs equal one plus a same-world resume, each rank's RNG included. | verified — every listed test passed (cpu evidence) | tests/test_ddp_resume.py::test_two_epochs_equal_one_plus_a_resume |
| CR-04 | stop at an update boundary + LAST checkpoint | cpu / fp32 / optuna | A mid-epoch stop leaves LAST, its tensors and the history at the previous committed epoch. | verified — every listed test passed (cpu evidence) | tests/test_search.py::test_a_mid_epoch_stop_leaves_the_last_commit_unrelabelled |
| CR-05 | on_train_end mutation + LAST refresh | cpu / fp32 / core | LAST holds the post-on_train_end weights; BEST keeps the training-time state. | verified — every listed test passed (cpu evidence) | tests/test_train_end_checkpoint.py::test_last_checkpoint_contains_on_train_end_mutationtests/test_train_end_checkpoint.py::test_best_checkpoint_stays_pre_mutation |
| CR-06 | DDP + changed world size or partition | cpu / fp32 / torch.distributed gloo | Refused before anything is restored. | unsupported — the negative test passes (cpu evidence) | tests/test_ddp_resume.py::test_a_changed_world_or_partition_fails_before_anything_is_restored |
| CR-07 | experiment plans + planned-horizon resume | cpu / fp32 / core | A plan stopped after epoch 1 of 4 and resumed to its plan equals the uninterrupted plan bit for bit. | verified — every listed test passed (cpu evidence) | tests/test_plan_execution.py::test_a_planned_plan_resume_equals_the_uninterrupted_plan_bit_for_bit |
| CR-08 | DDP + corrupted resume point | cpu / fp32 / torch.distributed gloo | The writer rank alone verifies the point; a corrupted one is refused on every rank with the same reason, before anything is restored. | unsupported — the negative test passes (cpu evidence) | tests/test_ddp_resume.py::test_a_corrupted_resume_point_is_refused_on_every_rank_naming_the_reason |
Model transformations
| ID | Features | Profile | Expected | Status | Evidence |
|---|---|---|---|---|---|
| MT-01 | torch.compile + QAT (topology-changing callback) | cpu / fp32 / torchao | Refused before the first batch, no run reserved. | unsupported — the negative test passes (cpu evidence) | tests/test_compile_execution.py::test_a_topology_changing_callback_is_refused_before_the_first_batch |
| MT-02 | torch.compile + checkpoint keys + eager reload | cpu / fp32 / core | Checkpoints hold the eager module's keys (no _orig_mod.) and reload eagerly with identical logits. | verified — every listed test passed (cpu evidence) | tests/test_compile_execution.py::test_checkpoints_hold_eager_keys_and_reload_eagerly |
| MT-03 | model surgery (deepen) + FeedFwdNN | cpu / fp32 / core | Deepening preserves the network's function. | verified — every listed test passed (cpu evidence) | tests/test_surgery_deepen.py::test_deepen_preserves_function_on_feed_fwd_nn |
| MT-04 | DDP + batch normalization | cpu / fp32 / torch.distributed gloo | Refused on every rank before any run (running statistics would differ per rank). | unsupported — the negative test passes (cpu evidence) | tests/test_ddp_adapter.py::test_preflight_refusals_fail_on_every_rank_before_any_run |
| MT-05 | torch.compile + custom train step or non-FP32 precision | cpu / fp32 / core | Refused up front; compile covers the built-in train step in FP32 on built-in non-graph nets. | unsupported — the negative test passes (cpu evidence) | tests/test_compile_execution.py::test_out_of_scope_requests_are_refused_up_front |
Inference and export
| ID | Features | Profile | Expected | Status | Evidence |
|---|---|---|---|---|---|
| IE-01 | KV-cache generation + full forward | cpu / fp32 / tokenizers | Cached and uncached generation produce the same output. | verified — every listed test passed (cpu evidence) | tests/test_generative_nn_model.py::test_kv_cache_produces_same_output_as_full_forward |
| IE-02 | ordered logits pipeline + KV cache | cpu / fp32 / tokenizers | A pipeline equals the explicit processor list, cached and uncached. | verified — every listed test passed (cpu evidence) | tests/test_ordered_logits_pipeline.py::test_generate_with_a_pipeline_equals_the_explicit_processor_list |
| IE-03 | ONNX export + module modes + gradients | cpu / fp32 / onnx, onnxscript | Export leaves parameters, gradients and mixed train/eval modes unchanged. | verified — every listed test passed (cpu evidence) | tests/test_inference_helpers_preserve_training_mode.py::test_to_onnx_keeps_parameters_gradients_and_mixed_modes |
| IE-04 | predict + training mode | cpu / fp32 / core | predict restores the caller's training mode, also after an exception. | verified — every listed test passed (cpu evidence) | tests/test_inference_helpers_preserve_training_mode.py::test_predict_restores_training_mode_after_calltests/test_inference_helpers_preserve_training_mode.py::test_predict_restores_training_mode_after_exception |
| IE-05 | ONNX Runtime execution + both exporters | cpu / fp32 / onnxruntime | Executed logits match the native model. | unverified — Owned by the dedicated export-conformance job (docs/export-conformance.md); this matrix does not certify ONNX Runtime parity on the CPU lane. | tests/test_export_conformance.py::test_both_exporters_match_native_logits_in_onnx_runtime |
| IE-06 | torch.compile + bounded benchmarks | cpu / fp32 / core | The eager and compiled sides start from identical weights and leave the caller's module unchanged; the compiled first call carries the compile cost, apart from warmed latency. | verified — every listed test passed (cpu evidence) | tests/test_benchmarking.py::test_compare_compile_starts_both_sides_from_identical_weightstests/test_benchmarking.py::test_compiled_first_call_carries_the_compile_cost |
| IE-07 | JEPA predictor under model.net + run bundles | cpu / fp32 / core | A real JEPA run's bundle rebuilds the encoder bit-exact without the predictor; without the option the error names it, and the rebuilt encoder refuses resume(). | verified — every listed test passed (cpu evidence) | tests/test_jepa_encoder_rebuild.py::test_a_jepa_run_bundle_rebuilds_its_encoder_bit_exacttests/test_jepa_encoder_rebuild.py::test_an_encoder_rebuilt_without_its_predictor_does_not_resume |
| IE-08 | JEPA predictor under model.net + Hugging Face Hub save | cpu / fp32 / huggingface_hub, safetensors | A Hub save with the predictor attached rebuilds the encoder bit-exact; every other key stays strict. | verified — every listed test passed (cpu evidence) | tests/test_jepa_encoder_rebuild.py::test_a_hub_save_with_the_predictor_attached_rebuilds_its_encodertests/test_jepa_encoder_rebuild.py::test_hub_exclusion_stays_strict_for_every_other_key |
Decisions and typed results
| ID | Features | Profile | Expected | Status | Evidence |
|---|---|---|---|---|---|
| DR-01 | Jev provider + decision jobs | cpu / fp32 / typesafe_sdk, httpx2 | A provider failure crossing a decision job surfaces as the job's typed cause with its request id; the job's questions share one request and nothing is retried; a sync-only client runs one call at a time. | verified — every listed test passed (cpu evidence) | tests/test_decision_jev.py::test_a_failure_crossing_a_decision_job_keeps_its_reason_and_request_idtests/test_decision_jev.py::test_a_sync_only_injection_is_sync_only_and_a_job_runs_it_one_call_at_a_time |
| DR-02 | Jev provider + decision benchmark | cpu / fp32 / typesafe_sdk, httpx2 | A provider failure crossing a benchmark collection keeps its typed reason and request id. | verified — every listed test passed (cpu evidence) | tests/test_decision_jev.py::test_a_failure_crossing_a_benchmark_keeps_its_reason_and_request_id |
| DR-03 | Result boundaries + decision providers | cpu / fp32 / core | A provider failure becomes an Err keeping its cause and request id, a malformed answer Err("invalid_decision_response") for any provider; an abstention is a success; other exceptions propagate. | verified — every listed test passed (cpu evidence) | tests/test_result_boundaries.py::test_a_provider_failure_is_an_err_keeping_its_cause_and_request_idtests/test_result_boundaries.py::test_a_malformed_answer_is_an_invalid_response_err_whatever_the_providertests/test_result_boundaries.py::test_abstention_is_a_success_with_probabilities_and_reasonstests/test_result_boundaries.py::test_other_provider_exceptions_propagate |
| DR-04 | Result boundaries + run bundles | cpu / fp32 / core | Inspecting a bundle never unpickles; a malformed bundle keeps its code, path and cause through composition. | verified — every listed test passed (cpu evidence) | tests/test_result_boundaries.py::test_inspecting_a_bundle_never_reaches_torch_loadtests/test_result_boundaries.py::test_a_malformed_bundle_keeps_code_path_and_cause_through_composition |
| DR-05 | decision-model pilot + model factory registry + checkpoint reload | cpu / fp32 / core | The pilot's model registers through register_model_factory with no new Nets member and reconstructs from its checkpoint with candidate-aligned records. | verified — every listed test passed (cpu evidence) | tests/test_decision_model_pilot.py::test_the_recipe_registers_through_feat_006_without_a_new_nets_membertests/test_decision_model_pilot.py::test_the_checkpoint_reconstructs_with_candidate_aligned_records_and_identities |
| DR-06 | decision-model pilot + compute cap + checkpoints | cpu / fp32 / core | The cap is never exceeded; a capped first epoch produces no checkpoint; exhaustion records a stop and keeps the artifacts. | verified — every listed test passed (cpu evidence) | tests/test_decision_model_pilot.py::test_the_cap_is_never_exceeded_and_a_capped_first_epoch_produces_no_checkpointtests/test_decision_model_pilot.py::test_budget_exhaustion_records_a_stop_and_keeps_the_artifacts |
| DR-07 | decision-model pilot + empirical claims | cpu / fp32 / core | The empirical mode reports blocked even with every resource present (the empirical training and baseline runs are not part of this repository); no empirical claim is made. | unsupported — the negative test passes (cpu evidence) | tests/test_decision_model_pilot.py::test_the_empirical_mode_with_every_resource_present_still_reports_blocked |
| DR-08 | Jev provider + Result boundaries | cpu / fp32 / typesafe_sdk, httpx2 | Each typed Jev failure becomes Err("provider_failure") and a malformed answer Err("invalid_decision_response"), both keeping the cause and request id; a success is Ok with candidate-aligned results. | verified — every listed test passed (cpu evidence) | tests/test_decision_jev.py::test_decide_result_turns_a_jev_failure_into_a_provider_failure_errtests/test_decision_jev.py::test_decide_result_reports_a_malformed_jev_answer_as_an_invalid_responsetests/test_decision_jev.py::test_decide_result_returns_aligned_jev_results |
Packaging and optional domains
| ID | Features | Profile | Expected | Status | Evidence |
|---|---|---|---|---|---|
| PK-01 | core install without domain extras + train / predict / save / reload | cpu / fp32 / core | With the vision, graph and plotting stacks hidden, a core install trains, predicts, saves and reloads, and lists only usable names. | verified — every listed test passed (cpu evidence) | tests/test_core_install.py::test_a_core_install_trains_predicts_saves_and_reloads_without_the_domainstests/test_core_install.py::test_a_core_install_lists_only_usable_names_and_pydoc_works |
| PK-02 | domain features + missing domain extras | cpu / fp32 / core | A domain name without its extra is refused, naming the extra, before data, model or random state is touched. | unsupported — the negative test passes (cpu evidence) | tests/test_core_install.py::test_a_domain_request_names_its_extratests/test_core_install.py::test_plot_features_refuse_before_running_the_modeltests/test_core_install.py::test_lr_finder_refuses_before_touching_data_model_or_rng |
Status meanings:
- verified: the listed tests assert the combination and passed.
- unsupported: refused, proven by an executed negative test (or an exact documented limitation).
- unverified: no executed evidence on the profiles CI runs (for example CUDA AMP on a CPU runner), or no current report.
- failed: a listed test did not pass or did not run.
Contributors: see CONTRIBUTING (scenario IDs to review when a boundary changes).