Numerical Values in Vertesaur - aarondandy/vertesaur GitHub Wiki

Computation with real numbers can be deceptively difficult. Calculations such as var c = a / b appear simple but the most common numerical representations choke on 1 / 3. For example (1-(1/3.0)) - (2/3.0) will be a non-zero value for most implementations. Regardless of if they are base two, ten, fixed point, or floating they will all suffer from the limitations of finite amounts of memory.

Early on I was taught various things regarding computation with real numbers that I feel are now incorrect due to my experience with this project. It is hoped that this document will both offer advice to others and serve as guidance when using or contributing to this project.

Contrary to "best practices" this project considers attempts to fight the imperfection of real number representations as potentially dangerous. It is instead recommended that you as a programmer accept that inputs and outputs may deviate from perfection.

When writing general purpose code without knowledge of the possible values it is strongly recommend that you not make assumptions. Avoid comparing values for equality using a distance tolerance such as an epsilon value. Furthermore if you are comfortable enough to compare values using greater-than or less-than operators you should also feel comfortable enough to perform a strict equality test.

Using the first example in the above image it is possible to calculate the intersection of two line segments so that the resulting location can't be represented using the current numerical representation. Instead the intersecting point would be moved to the nearest representable coordinate that may not even intersect both lines or even any of the lines in some cases.

One possible solution as shown in the second example of the above image is to change the problem so that the location of the intersection occurs over a representable location. In this example each of the two line segments are split in two so they can be "bent" into the correct position. Ultimately the result will be imprecise. At least this way you don't have a resulting intersection point that does not intersect both inputs.

This is the most important rule: Do not attempt to "fix" floating point. Instead think about how you can change the problem to better tolerate imprecision.

An odd quirk with .NET that may take you by surprise is that System.Double (or double) is faster than System.Single (or float) for arithmetic. Yes really, I promise! To be simple about it, floats get converted to doubles when you do any arithmetic so they get the extra overhead of conversion. So don't use float and claim it to be in the name of performance as you will make it slower while reducing precision.

Multiple times I have run into issues with an implementation and been told that I should have used fixed point. I never got an explanation why or even an example and I now believe that to be unhelpful advice.

While there are many numbers that can't be represented by floating point there are also plenty of numbers that can't be represented by fixed point. Ultimately I worked around those issues not by changing my numerical representations but instead by changing my approach to the problem entirely. See the most important rule above.