Performance Tests - nolansingroy/cocos2d-x-1 GitHub Wiki

How to run performance tests

Basic rules

  • Never run the tests on a Simulator and/or Emulator. Always use real devices
  • The older the device the better
  • Never run the tests in DEBUG mode. Always use RELEASE mode
  • Double check that RELEASE mode is using all the possible optimizations for the C++ code

Devices are multitasking, and many tasks are run in the background and those tasks could affect the performance. In order to minimize that do:

  • Turn Airplane mode ON
  • Disable Wifi
  • Disable Bluetooth
  • Kill all running tasks / applications

Logging

  • Performance Tests must be run before releasing an stable version
  • The results must be logged in this spreadsheet
  • In order to know the performance of the new release, a comparison must be done with the previous versions. The comparison must try to use the same testing environment. As an example:
    • Same devices (MUST)
    • Same device operating system (SHOULD)
    • Same compiler and toolchain (DESIRABLE)

Starting from Cocos2d-x v3.3, the "auto run" feature of Sprite Performance Tests logs in the console the output of the tests. This output can be copy & pasted into the spreadsheet

Measuring the performance

Most of the Cocos2d-x performance tests relies on watching the FPS number. Although this is not a reliable way to test how fast a particular component is, it gives us at least a way to know how fast Cocos2d-x is in general.

Note: The method of taking a look at the FPS MUST NOT be used to measure the performance of a particular algorithm, or function. Instead Sampling Profiling like Xcode Instruments, or Linux Perf must be used. Intrusive Profiling is also valid method, like using the CC_PROFILER_START and CC_PROFILER_STOP macros.

Looking at the FPS manually

Pros:

  • Doesn't require additional services / tools:
    • Wifi can be turned off
    • Xcode / Visual Studio are not needed
  • Can be used to test any kind of tests:
    • eg: Sprite Test with 5250 sprites

Cons:

  • Like any other manual process, it is error prone and takes a lot of time.

Suggested for:

  • Only to test a few tests. Usually to make sure that the Automated test output is correct.

Automated Tests

As of v3.3 only the Sprite Performance Test can be automated. The rests of the test must use the "manual" method.

In order to run the automated version of Sprite Performance Test, you have to:

  1. Launch cpp-tests
  2. Select the Performance tests test
  3. Select the Sprite Perf Test subtest
  4. Press the Auto Test Off button

sprite auto test

Getting the results: cocos2d console

The automated tests logs the results using the cocos2d::Console::log() function. That means that the cocos2d::Console has the output. And it is possible to get the output by connecting the cocos2d::Console by using telnet (by default cpp-tests has the the cocos2d::Console enabled):

$ telnet 10.0.1.16 5678

10.0.1.16 would the IP address of the device.

Once inside the console, you can type help to see all the available commands:

console help

And in order to redirect the log() messages to the console, you must type:

> debugmsg on

Once the automated test is finished, the output will be in the console, and you will see something like the following:

console output

The result can be copy & pasted into the spreadsheet

Pros:

  • Is very reliable, specially if the tests does not use the log() function during the tests

Cons:

  • Requires WiFi to be ON, violating one of the Basic Rules described above.
  • Requires to touch the Phone every now and then to prevent it from going to sleep (this can be easily fixed in the tests, but as of v3.3 it was not fixed)

Suggested for:

  • Use this method for automated tests (and fix the "go to sleep bug"!)

Getting the results: Xcode / Visual Studio / Instruments

It is possible to use Xcode or Visual Studio to redirect the output of the cocos2d::Console to Xcode's or Visual Studio's.

Although in theory these tools should not affect the performance of the tests, using Xcode's console proved to make the tests slower (Tested on Xcode 6.1 connected to an iPod Touch 4 using iOS 6.1.6). No penalty was found on Visual Studio at the moment.

xcode console

Pros:

  • Can be used with everything turned off (Wifi). They only require to connect to the device using the USB cable.
  • They keep the phone awake preventing it from going to sleep mode

Cons:

  • Xcode v6.1 + iPod Touch 4 with iOS 6.1.6 makes some tests to run slower

Suggested for:

  • Avoid this method since it is not clear why Xcode make the tests run slower.