Measuring Performance - Falmouth-Games-Academy/comp350-research-journal GitHub Wiki
Performance in games
Frames Per Second (FPS) is very often used to measure the performance of a game. While FPS is a good indicator of performance, it is inconsistent when making comparisons. A much more accurate way of measuring overall performance is milliseconds(ms) per frame. For example, if a game is running at 60fps (16.666ms per frame), and after some optimisation, it is now running at 80fps (12.5ms per frame), that is a mere 4.166ms per frame improvement. On the other hand, if a game is running at 20fps (50ms per frame), and is now running at 40fps (25ms per frame), that is a 25ms per frame boost! 1(https://engineering.riotgames.com/news/profiling-measurement-and-analysis). Unfortunately, out of the box games rarely provide a settings option to measure the time taken to complete a frame. FPS counters, on the other hand, are found in most modern games. Profilers can be used to measure both the FPS and the ms per frame, see Profiling for more details.
Performance of Algorithms
To measure the performance of an algorithm the amount of time it takes to complete from start to finish (Time Complexity) is often used 2(https://www.toolsqa.com/data-structures/big-o-notation/). This will show how fast the algorithm is and can be compared to other algorithms that achieve the same outcome. The less time the algorithm has taken, the better.
Space Complexity can also be used to measure the performance of an algorithm 2(https://www.toolsqa.com/data-structures/big-o-notation/). The less the memory that an algorithm uses the better it's performance.
Time Complexity is often more valued than Space Complexity. Daniel R.Page suggests that this is because we can re-use memory, however, we cannot re-use time 3(https://www.quora.com/Why-is-time-complexity-discussed-more-frequently-than-space-complexity). Dhavel Dave offers a different opinion, he argues that it depends on the goals and the data the system/algorithm has to handle 4(https://www.quora.com/What-matters-a-lot-space-complexity-or-time-complexity-Is-it-good-thing-that-minimizing-time-complexity-by-increasing-space-complexity).
Both of Time Complexity and Space Complexity are ways of measuring performance and both use Big O notation to denote how efficient the algorithm is when the input scale.
When to measure?
When considering the fitness score of a complex system, we must decide what to measure and when. Profiling to discover where the majority of the computation time is being spent according to The Pareto principle (see Profiling), requires measuring the performance of specific expensive subsystems, whereas measuring performance to optimise towards particular constraints, such as Mobile device optimisation, will require a more broad measuring procedure to fit size or performance requirements. Approaches such as average-best-of, and best-so-far are invaluable to differentiate between priorities when measuring performance 5(https://pdfs.semanticscholar.org/daa5/6b189b8484e3e136bfb38b2efadb70736ca3.pdf).
References
[1] https://engineering.riotgames.com/news/profiling-measurement-and-analysis
[2] https://www.toolsqa.com/data-structures/big-o-notation/
[3] https://www.quora.com/Why-is-time-complexity-discussed-more-frequently-than-space-complexity
[5] Ben-Romdhane, Hajer, Enrique Alba, and Saoussen Krichen. "Best practices in measuring algorithm performance for dynamic optimization problems." Soft Computing 17.6 (2013): 1005-1017.
[6] https://software.intel.com/en-us/articles/unreal-engine-4-optimization-tutorial-part-1