Testing - jamescourtney/FlatSharp GitHub Wiki

FlatSharp takes quality seriously. To that end, FlatSharp is tested in a variety of overlapping ways:

  • Oracle Testing: Verifies that FlatSharp can interoperate with the official Google FlatBuffer library. Note that this does not mean that the two libraries emit the same bytes for the same schema; FlatBuffers allows a great deal of flexibility for valid encodings of the same data.

  • Byte Level Output Testing: Verifies that FlatSharp emits the specific sequence of bytes expected for a given schema. These tests ensure that FlatSharp is not writing junk data, is performing alignment correctly, that vtables are properly deduplicated, and that other nuances of the binary format are tested in a robust fashion.

  • Byte Level Input Testing: Verifies that FlatSharp correctly handles a sequence of sometimes invalid input bytes. Tests like these ensure that omitted required fields result in an error, malformed data results in an exception, etc.

  • Higher Level Testing: Verifies that the various deserialization modes behave as expected, the FlatSharp.Runtime library performs as expected, schema validations work, etc.

  • Mutation Testing: Uses Stryker Mutator to inject bugs into code generated by the FlatSharp compiler and verify that a special set of tests are sensitive to these regressions. The mutation tests are extremely exhaustive, but only run on a significant subset of FlatSharp's functionality. At least 95% of injected bugs (mutants) must be caught by the tests to pass.

  • Code Coverage: Measures the total code coverage of FlatSharp's tests. While Code Coverage alone is not an excellent measure, combined with Mutation Testing it is an indicator of comprehensiveness. Code coverage is measured on each PR. PRs must increase code coverage to pass.

  • Performance Testing: FlatSharp maintains a suite of microbenchmarks for comparing the current version to the previous one. These help ensure that FlatSharp's performance doesn't regress from release to release. The microbenchmarks are run on an ad-hoc basis.

  • Legacy C# Compilation: While FlatSharp itself is written using the latest language versions, it emits C# 8.0 code. FlatSharp validates that generated code compiles against this target at PR time.

  • AOT Testing: FlatSharp validates .NET NativeAOT compatibility at CI time by running the full suite of unit tests using NativeAOT-compatible MSTest runners on MacOS, Windows, and Linux.

  • Mono Testing: FlatSharp has a small number of "it's not totally broken" tests for Mono scenarios. These cover both AOT and JIT compilation. These run only on Linux.