1. Purpose and Scope
2. Validation Philosophy
3. Floating-Point Fundamentals
4. Precision and Data Types
5. Error Measurement
6. Absolute and Relative Tolerances
7. Reference Implementations
8. Tensor Arithmetic Validation
9. Broadcasting Validation
10. Matrix Multiplication Validation
11. Reduction Validation
12. Automatic Differentiation Validation
13. Finite-Difference Gradient Checking
14. Gradient Accumulation Validation
15. Activation Function Validation
16. Loss Function Validation
17. Optimizer Validation
18. Neural Network Layer Validation
19. Convolution and Pooling Validation
20. Normalization Validation
21. Recurrent and Attention Validation
22. Numerical Stability
23. Mixed Precision Validation
24. CUDA Numerical Validation
25. Quantization Validation
26. ONNX Interoperability Validation
27. Serialization Integrity
28. Property-Based and Metamorphic Testing
29. Regression Testing
30. Validation Reporting
31. Acceptance Criteria
32. Common Numerical Mistakes
33. Official References

Numerical validation establishes whether a computational implementation produces results consistent with its mathematical specification.

For EveryO, this includes:

• Tensor arithmetic.
• Matrix operations.
• Reductions.
• Automatic differentiation.
• Neural network layers.
• Loss functions.
• Optimizer updates.
• Mixed precision.
• CUDA kernels.
• ONNX exports.
• Model serialization.

The validation process should answer:

1. Is the mathematical operation implemented correctly?
2. Are output shapes and dtypes correct?
3. Are gradients consistent with the forward computation?
4. Are numerical errors within justified tolerances?
5. Are results reproducible?
6. Do optional backends produce compatible results?

         DEFINE OPERATION
                |
                v
       ESTABLISH REFERENCE
                |
                v
         GENERATE INPUTS
                |
                v
         COMPUTE RESULTS
                |
                v
         MEASURE ERRORS
                |
                v
       APPLY ACCEPTANCE RULE
                |
          +-----+-----+
          |           |
          v           v
         PASS        FAIL
          |           |
          v           v
       DOCUMENT    INVESTIGATE

Performance improvements must preserve the expected numerical behavior.

Use an independent implementation or analytically derived result whenever possible.

Test:

• Typical values.
• Boundary values.
• Extreme magnitudes.
• Degenerate shapes.
• Randomized inputs.
• Invalid inputs.

Define tolerances before examining results.

A statement such as "99.99% accurate" is generally not meaningful without specifying:

• The measured quantity.
• The error metric.
• The reference.
• The test distribution.
• The acceptance threshold.

Computers represent real numbers using finite precision.

A common model is:

[ fl(x \circ y)=(x\circ y)(1+\delta) ]

where:

[ |\delta|\leq u ]

under suitable assumptions, with (u) representing unit roundoff.

Floating-point addition is not generally associative.

[ (a+b)+c \neq a+(b+c) ]

in all floating-point cases.

Subtracting nearly equal numbers can destroy significant digits.

A result larger than the representable range may become infinity.

Very small values may lose precision or become zero.

These values are approximate and depend on the magnitude and operation.

float64

Useful for numerical reference calculations and gradient checking.

float32

Common baseline for neural network computation.

float16

Useful for selected mixed-precision workloads, with increased numerical risk.

NumPy can report floating-point machine epsilon:

import numpy as np

print(np.finfo(np.float16).eps)
print(np.finfo(np.float32).eps)
print(np.finfo(np.float64).eps)

Machine epsilon is not a universal acceptable error threshold for complex computations.

[ E_{\mathrm{abs}}=|x_{\mathrm{actual}}-x_{\mathrm{reference}}| ]

[ E_{\mathrm{rel}}= \frac{|x_{\mathrm{actual}}-x_{\mathrm{reference}}|} {|x_{\mathrm{reference}}|} ]

Relative error is undefined when the reference is zero.

A practical alternative is:

[ E_{\mathrm{rel,stable}}= \frac{|x_{\mathrm{actual}}-x_{\mathrm{reference}}|} {\max(\epsilon,|x_{\mathrm{reference}}|)} ]

[ E_{\max}=\max_i|a_i-r_i| ]

[ RMSE= \sqrt{\frac{1}{N}\sum_{i=1}^{N}(a_i-r_i)^2} ]

Reference:

[ r=10.0 ]

Actual:

[ a=10.002 ]

Absolute error:

[ E_{\mathrm{abs}}=0.002 ]

Relative error:

[ E_{\mathrm{rel}}=0.0002 ]

Percentage relative error:

[ 0.02% ]

This is an illustrative calculation, not an EveryO measurement.

A common elementwise acceptance rule is:

[ |a-r|\leq atol+rtol|r| ]

where:

• a is the actual result.
• r is the reference.
• atol is absolute tolerance.
• rtol is relative tolerance.

import numpy as np

np.testing.assert_allclose(
    actual,
    reference,
    rtol=1e-5,
    atol=1e-7,
)

Here, actual and reference are arrays prepared by the test.

Tolerance should depend on:

• Dtype.
• Operation count.
• Numerical conditioning.
• Expected output magnitude.
• Reference precision.

Do not increase tolerances solely to make failing tests pass.

Investigate the source of the discrepancy first.

NumPy is a useful reference for tensor arithmetic and matrix operations.

For simple functions, derive expected outputs mathematically.

Optional TensorFlow comparisons can help validate supported neural network operations.

ONNX Runtime can validate exported model execution.

Because EveryO uses NumPy internally, agreement with NumPy does not always provide fully independent evidence.

Where practical, combine reference comparisons with analytical tests and mathematical invariants.

[ C=A+B ]

Example:

import numpy as np
import everyo as eo

a_np = np.array([1.0, 2.0, 3.0])
b_np = np.array([4.0, 5.0, 6.0])

a = eo.tensor(a_np)
b = eo.tensor(b_np)

actual = a + b
expected = a_np + b_np

np.testing.assert_allclose(
    actual.data,
    expected,
)

Validate:

[ C=A\odot B ]

where (\odot) represents elementwise multiplication.

Include nonzero denominators and test documented behavior for zero denominators.

Test values with similar magnitudes to investigate cancellation.

Broadcasting extends compatible dimensions during elementwise operations.

A shape: (4, 3)
B shape: (1, 3)

Output shape: (4, 3)

• Scalar broadcasting.
• Row broadcasting.
• Column broadcasting.
• Higher-dimensional broadcasting.
• Invalid shape combinations.

Broadcasting backward passes must correctly reduce gradients over expanded dimensions.

Forward results and gradients must match justified references.

[ C_{ij}=\sum_{k=1}^{K}A_{ik}B_{kj} ]

import numpy as np
import everyo as eo

a_np = np.array([
    [1.0, 2.0],
    [3.0, 4.0],
])

b_np = np.array([
    [5.0, 6.0],
    [7.0, 8.0],
])

actual = eo.matmul(
    eo.tensor(a_np),
    eo.tensor(b_np),
)

expected = a_np @ b_np

np.testing.assert_allclose(
    actual.data,
    expected,
)

Expected result:

[[19. 22.]
 [43. 50.]]

• Rectangular matrices.
• Batched matrix multiplication.
• Different magnitudes.
• Zero matrices.
• Identity matrices.
• Gradient correctness.

Common reductions include:

• Sum.
• Mean.
• Maximum.
• Minimum.

[ S=\sum_{i=1}^{N}x_i ]

[ \mu=\frac{1}{N}\sum_{i=1}^{N}x_i ]

• Entire-tensor reduction.
• Axis-specific reduction.
• Singleton dimensions.
• Large input arrays.
• Positive and negative values.

Different reduction orders may produce slightly different floating-point results.

EveryO uses reverse-mode automatic differentiation.

For:

[ y=f(g(x)) ]

the derivative is:

\frac{df}{dg}\frac{dg}{dx} ]

[ y=x^2 ]

Then:

[ \frac{dy}{dx}=2x ]

import numpy as np
import everyo as eo

x = eo.tensor(
    [3.0],
    requires_grad=True,
)

y = x * x

y.backward()

np.testing.assert_allclose(
    x.grad,
    np.array([6.0]),
)

• Correct gradient values.
• Correct gradient shapes.
• Chain rule.
• Shared graph branches.
• Broadcasting.
• Reduction gradients.

Finite differences provide a numerical approximation of derivatives.

[ f'(x)\approx \frac{f(x+h)-f(x-h)}{2h} ]

For:

[ f(x)=x^3 ]

the analytical derivative is:

[ f'(x)=3x^2 ]

At (x=2):

[ f'(2)=12 ]

def f(x):
    return x ** 3


x = 2.0
h = 1e-5

numeric = (
    f(x + h) - f(x - h)
) / (2 * h)

analytic = 3 * x ** 2

print("Numeric:", numeric)
print("Analytic:", analytic)
print("Error:", abs(numeric - analytic))

If (h) is too large, truncation error increases.

If (h) is too small, floating-point cancellation may dominate.

Use float64 references where practical and avoid nondifferentiable points.

When a variable contributes to multiple graph branches, gradient contributions must be accumulated.

[ y=x^2+x^3 ]

Then:

[ \frac{dy}{dx}=2x+3x^2 ]

At (x=2):

[ \frac{dy}{dx}=4+12=16 ]

import numpy as np
import everyo as eo

x = eo.tensor(
    [2.0],
    requires_grad=True,
)

y = (x * x) + (x * x * x)

y.backward()

np.testing.assert_allclose(
    x.grad,
    np.array([16.0]),
)

• Shared parameters.
• Repeated operations.
• Multiple branches.
• Gradient resets between independent steps.

[ ReLU(x)=\max(0,x) ]

Test negative, zero, and positive inputs.

[ \sigma(x)=\frac{1}{1+e^{-x}} ]

Test:

• Large positive inputs.
• Large negative inputs.
• Inputs near zero.

[ \tanh(x) ]

Validate output bounds and derivatives.

[ softmax(x_i)= \frac{e^{x_i}}{\sum_j e^{x_j}} ]

Verify:

[ \sum_i softmax(x_i)\approx1 ]

Softmax implementations should avoid unnecessary overflow from large exponentials.

[ MSE= \frac{1}{N}\sum_{i=1}^{N}(y_i-\hat{y}_i)^2 ]

[ MAE= \frac{1}{N}\sum_{i=1}^{N}|y_i-\hat{y}_i| ]

For a one-hot target distribution:

[ L=-\sum_i y_i\log(p_i) ]

• Forward values.
• Reduction behavior.
• Target shape handling.
• Gradient correctness.
• Extreme input behavior.

Where supported, prefer loss implementations designed to accept logits directly rather than manually combining potentially unstable operations.

Optimizers should be tested against independently calculated parameter updates.

[ \theta_{t+1}= \theta_t-\eta g_t ]

Given:

[ \theta_t=2.0 ]

[ g_t=0.5 ]

[ \eta=0.1 ]

Then:

[ \theta_{t+1}=1.95 ]

• Parameter updates.
• Momentum state.
• Bias correction.
• Weight decay behavior, if supported.
• Optimizer state restoration.

Validate several consecutive updates, not just the first step.

[ Y=XW+b ]

Validate:

• Output shape.
• Forward values.
• Weight gradients.
• Bias gradients.
• Input gradients.

Verify correct composition of layer outputs.

Confirm trainable parameters are exposed to the optimizer.

Verify training and evaluation behavior for applicable layers.

Validate:

• Kernel application.
• Padding.
• Stride.
• Channel ordering.
• Output dimensions.
• Backward gradients.

EveryO's documented image layout is:

(batch, height, width, channels)

Validate maximum and average pooling against small analytically tractable examples.

Test:

• Odd dimensions.
• Non-square images.
• Different strides.
• Different padding.
• Multiple channels.

Normalization layers require careful numerical testing.

Verify:

• Mean calculation.
• Variance calculation.
• Epsilon handling.
• Running statistics.
• Training/evaluation differences.

Verify normalization over the intended dimensions.

Small variance values can cause unstable division without suitable epsilon handling.

Compare gradients against finite differences for small inputs.

Validate:

• Sequence dimensions.
• Hidden-state updates.
• Initial states.
• Variable sequence lengths.
• Backward propagation through time.

Check gate computations and state transitions against reference implementations where available.

Scaled dot-product attention uses:

[ Attention(Q,K,V)= softmax\left( \frac{QK^T}{\sqrt{d_k}} \right)V ]

• Attention scores.
• Scaling.
• Softmax normalization.
• Mask handling, where supported.
• Output dimensions.
• Gradients.

Validate each subcomponent before testing the complete block.

• Overflow.
• Underflow.
• Catastrophic cancellation.
• Division by zero.
• Invalid logarithms.
• Non-finite gradients.

A common technique is:

[ softmax(x_i)= \frac{e^{x_i-m}} {\sum_j e^{x_j-m}} ]

where:

[ m=\max_j x_j ]

m+\log\left(\sum_i e^{x_i-m}\right) ]

Include extreme but valid inputs.

When a non-finite result appears, identify the first operation that produced it.

Determine whether mixed-precision execution produces acceptable numerical results.

Use float32 or float64 execution where appropriate.

• Forward outputs.
• Loss values.
• Gradients.
• Optimizer updates.
• Final model metrics.

Verify that loss scaling behaves as documented and handles non-finite gradients appropriately.

Mixed precision can change numerical behavior and is not universally faster.

Compare supported CUDA operations against CPU reference results.

• Small arrays.
• Large arrays.
• Boundary dimensions.
• Non-power-of-two dimensions.
• Different input magnitudes.
• Repeated execution.

import everyo as eo

print(eo.cuda.is_available())
print(eo.cuda.unavailable_reason())

CUDA benchmarks must verify numerical correctness before reporting performance.

Actual CUDA runtime correctness requires testing on compatible hardware.

A successful compilation does not establish correct device execution.

Measure the numerical effects of reduced-precision weight representation.

[ E_q=|w-\hat{w}| ]

where:

• (w) is the original weight.
• (\hat{w}) is the reconstructed or quantized approximation.

Compare:

• Model outputs.
• Prediction changes.
• Evaluation metrics.
• Model size.

EveryO's documented dynamic quantization uses int8 weights but does not provide specialized faster int8 execution kernels.

Therefore, reduced model size should not be confused with guaranteed inference acceleration.

Verify that exported ONNX models preserve the intended behavior.

1. Construct a supported model.
2. Generate controlled input data.
3. Run EveryO inference.
4. Export to ONNX.
5. Run ONNX Runtime.
6. Compare outputs.
7. Record numerical differences.

import everyo as eo

model = eo.Sequential(
    eo.Linear(4, 8),
    eo.ReLU(),
    eo.Linear(8, 2),
)

model.eval()

eo.export_onnx(
    model,
    "validation.onnx",
    input_shape=(1, 4),
)

Verify the exported model's supported input dimensions.

Traced graphs may preserve a fixed execution path or shape assumptions.

Do not assume arbitrary dynamic shapes are supported.

Verify that saving and restoring a model preserves its supported state and behavior.

import everyo as eo

eo.save(model, "validation.evo")

restored = eo.load("validation.evo")

Compare:

• Parameter shapes.
• Parameter values.
• Model outputs.
• Relevant configuration.

EveryO's .evo format is designed to avoid pickle-based execution.

However, untrusted model archives should still be treated as potentially malformed or resource-intensive.

Property-based testing checks mathematical relationships across many inputs.

For suitable matrices:

[ AI=A ]

where (I) is the identity matrix.

[ A+0=A ]

For elementwise addition:

[ A+B=B+A ]

within appropriate floating-point tolerances.

[ (A^T)^T=A ]

For compatible data:

[ mean(x)\approx\frac{sum(x)}{N} ]

Floating-point implementations may not satisfy algebraic identities exactly because of rounding.

Use justified tolerances.

Prevent previously corrected numerical defects from recurring.

        IDENTIFY DEFECT
              |
              v
       CREATE MINIMAL TEST
              |
              v
       CONFIRM TEST FAILS
              |
              v
         IMPLEMENT FIX
              |
              v
        CONFIRM TEST PASSES
              |
              v
         RUN FULL SUITE
              |
              v
         RETAIN TEST

Include:

• Original failing input.
• Related boundary cases.
• Numerical tolerances.
• Relevant dtype variations.

Run applicable numerical tests in continuous integration.

Optional hardware tests should clearly report skips when hardware is unavailable.

# Numerical Validation Report

## Operation

Name:

## Environment

- EveryO version:
- Python version:
- NumPy version:
- Operating system:
- Device:
- Dtype:

## Test Configuration

- Input shape:
- Input distribution:
- Random seed:
- Reference implementation:

## Error Metrics

| Metric | Value |
|---|---|
| Maximum absolute error | |
| Maximum relative error | |
| RMSE | |
| Absolute tolerance | |
| Relative tolerance | |

## Result

Pass / Fail / Not Evaluated

## Limitations

Describe any limitations.

## Reproduction

Provide exact commands.

Do not omit failed or skipped tests.

Include sufficient details to reproduce the validation.

A numerical validation task should be considered complete when:

• The mathematical specification is clear.
• A suitable reference is available.
• Input shapes are documented.
• Dtypes are documented.
• Numerical tolerances are justified.
• Forward outputs are verified.
• Gradients are verified where applicable.
• Boundary cases are tested.
• Non-finite behavior is investigated.
• Relevant hardware is identified.
• Regression tests are retained.
• Results are reproducible.
• Limitations are disclosed.

All required checks satisfy their defined acceptance criteria.

At least one required check violates its acceptance criteria.

The test was not executed, for example because required hardware was unavailable.

Not Evaluated must never be reported as Pass.