Running the worker: overview - xoto10/fishtest GitHub Wiki

In case you have not done so already, get a new account/password registering on fishtest:

https://tests.stockfishchess.org/signup

Please mind that after the registration you will be automatically redirected to the login page. The message "Permission Required. Creating or modifying tests requires you to be logged in. If you don't have an account, please Register." is an information, not an error message.

You can test how the fishtest worker works using a free 2 cores instance for max 6 hours on Google Colaboratory:

• signup for a free account
• create a new notebook
• paste this text writing your fishtest account/password:

!git clone --single-branch --branch master https://github.com/glinscott/fishtest.git
!python3 fishtest/worker/worker.py <username> <password> --use_all_cores True

• start the notebook
• you can save the notebook on your Google Drive

In case you have not installed the worker on your computer yet, follow the installation instructions on these pages :

• Running the worker on Windows
• Running the worker on Linux
• Running the worker on Macintosh
• Running the worker in the Amazon AWS EC2 cloud

To launch the worker using the default parameters (e.g. using 3 CPU cores) simply open a console window in worker directory and run the following command providing your fishtest username and password:

python3 worker.py username password

Add the option concurrency to control the number of CPU cores allocated to the worker. The safest max setting suggested is to use the number of physical cores leaving one core for the OS. On a PC dedicated only to fishtest it's possible to set concurrency to the number of virtual cores leaving one core for the OS, but with the number of workers contributed today to the framework this is not strictly necessary.

python3 worker.py username password --concurrency 7

At the start the workers writes the parameters into the configuration file fishtest.cfg, so next time you can start the worker with the simple:

python3 worker.py

On Linux and Macintosh, you can use the nohup command to run the worker as a background task.

nohup python3 worker.py &

python3 worker.py <username> <password> [-P <URL protocol>] [-n <URL domain>] [-p <URL port>] [-c <number of cores>] [-m <memory>] [-t <test min thread number>] [-f <fleet flag>] [-a <all cores flag>] [-w <write config flag>]

<username>         : your username on fishtest, first argument
<password>         : your password on fishtest, second argument
-h / --help        : show the help message and exit
-P / --protocol      PROTOCOL      : protocol of the fishtest server URL (string, default: https)
-n / --host          HOST          : address of fishtest server URL (string, default: tests.stockfishchess.org)
-p / --port          PORT          : port of fishtest server URL (number, default: 443)
-c / --concurrency   CONCURRENCY   : the number of core allocated to the worker (number, default: 3)
-m / --max_memory    MAX_MEMORY    : memory used by the worker (number, default: system memory/2 expressed in MB)
-t / --min_threads   MIN_THREADS   : minimum number of thread that a test must have to be assigned to the worker (number, default: 1)
-f / --fleet         FLEET         : stop the worker when fishtest has no task to do (boolean, default: False)
-a / --use_all_cores USE_ALL_CORES : use all the CPU online cores (boolean, default: False)
-w / --only_config   ONLY_CONFIG   : write the config file and exit (boolean, default: False)

Currently, the game-playing program (cutechess-cli) used by the worker will flag games as lost on time in case of a very large (>32) concurrency for very fast games. Hardware with more than 32 cores will therefore be assigned to tests running at long time control (LTC). In very rare cases, no LTC tests run on fishtest, in which case the worker might be idle.

workers with very many threads (e.g. 100) might exceed some of the defaults limits on a linux server. Ensure that max user processes (ulimit -u) and open files (ulimit -n) are sufficiently large.

To stop gracefully the worker write a file named fish.exit near the worker.py script, the worker will stop after finishing the batch games. On linux simply run the command touch fish.exit. Remember to delete the fish.exit before the next start or the worker will exit immediately.

By the way you can stop as soon as possible the worker simply killing the process, e.g. CRTL+C in the command window, using the kill command on linux etc. The games played by your worker will be used by the framework, but the total batch games number will be credited to the user that will complete your batch.

The worker makes some GitHub API requests. GitHub sets these rate limits:

• 60 requests/hour for single IP address for unauthenticated requests
• 5000 requests/hour for authenticated requests

The lower rate is perfectly fine for the majority of CPU contributors, to switch to the higher rate:

• login on your GitHub account (sign up for a free account on GitHub)
• from the token creation page create a new authorization token enabling the "public_repo" scope, copy the <personal-access-token> number
• create a text file:
  • Linux and Windows Subsystem for Linux: touch ${HOME}/.netrc && chmod 600 ${HOME}/.netrc
  • Windows: C:\Users\<Your_Username>\.netrc or C:\Users\<Your_Username>\_netrc (check your $HOME)
• write this content:

machine api.github.com
login <personal-access-token>
password x-oauth-basic

or

machine api.github.com
login <your_github_username>
password <personal-access-token>

• on Windows be sure to delete the .txt extension (_netrc, not _netrc.txt)