Scaling - guydavis/machinaris GitHub Wiki

Machinaris' primary goal is to be an all-one-solution that allows easy farming and plotting through control and monitoring of your entire farm.

  • For many, their entire farm may be a single Docker container running just the Chia blockchain on a bit of spare drive space. Check out scaling down below to save memory.
  • Others may add workers farming the same plots to additional forks such as Flax and HDDCoin.
  • They add additional machines harvesting more plots.
  • They may increase their overall plotting speed adding dedicated plotter systems, controlled via the same single WebUI.

Scaling Up

Eventually, they may scale up their operation to be a deployment on hundreds workers (forks), harvesting many tens of thousands of plots, totaling many petabytes of storage. Not all of us will grow our farms this much, but Machinaris is fully able to scale alongside your farm if you do.

The following are some settings to tune as your farm gets very large, to ensure it's smooth running. Settings are grouped by mode of worker (full-node, harvester, plotter).

Full-Nodes

By default, status is gathered roughly every 2 minutes. This works well for smaller farms, but very large farms need a slower pace. Edit the config file at:

  • ~/.machinaris[-FORK]/machinaris/config/api.cfg to hold STATUS_EVERY_X_MINUTES = 10

On the Chia fullnode (aka Machinaris controller), by default, challenges are shown for the last 15 minutes. For a large number of harvesters, this can slow the Index page chart down. To narrow the time range:

  • ~/.machinaris/machinaris/config/web.cfg to hold MAX_CHART_CHALLENGES_MINS = 5

Restart the full-node container after changing these settings.

As well, the automatic Plots Check and Plots Analyze feature is nice to have, but not be what you want if you have a huge number of plots. You can optionally disable this checking to lessen resource usage.

As well, one may optionally disabling Forktools, which can prevent plot loading issues, in first 15 minutes of container launch, by the Chia harvester on extremely large farms.

Harvesters

The default Chia setting to check for new plots appearing on disks is once every two minutes. For harvester (and full-node) systems with many drives and many plots, this can mean challenges do not get responded to in a timely fashion. Machinaris, thru the Forktools utility, increases this refresh rate to once every 30 minutes. However, on systems which are "fully-plotted", not regularly gaining new plot files, this can be increased even further to once every 5 hours or more.

To change this setting, visit the Settings | Tools page of the Machinaris WebUI, and increase the SETPLOTLOADFREQUENCY value. The blockchain services are restarted when you save. This change is persisted across container restarts.

Plotters

A common choice is to make a worker both a harvester and a plotter, in case you might one day wish to plot with it. However, if you have a large number (10+) of workers who are 100% harvesters and never plot on them, but you chose mode=harvester,plotter then each of these workers must continually report into the controller that no plotting jobs are happening. This is unnecessary database load on the controller. In this case, I strongly recommend setting the worker mode to just harvester. Only set mode to harvester,plotter if you are actually plotting with the worker.

Scaling Down

For those running many Machinaris containers (one per blockchain), here are your options to save memory:

  1. Fullnodes run the wallet process 24/7 by default. This consumes about 1/3 of total memory for a blockchain fullnode. However, you don't need to have the wallet always running in order to farm. Therefore, you can choose to run the Wallet just once a day, for a few minutes to sync. When rotated around all blockchain fullnodes, this daily wallet sync approach can save 1/3 of memory.
  2. Fullnode Chia blockchain (and forks) default to assuming they each have full rights to all resources on the entire system. For example, when running many (20+) blockchain fullnodes on a machine with 24 cores and 48 GB memory, each blockchain will launch numerous full_node_worker processes, wasting memory. In that case, be sure to set the Forktools setting SETFNRESERVEDCORES.
  3. Some blockchain fullnodes are still running forks of the Chia codebase from 2021, missing out many resource efficencies added by Chia in 2022. For those old/stale/defunct blockchain forks, expect to chew 4 GB - 6 GB for a fullnode. I recommend running only fork blockchains with active communities and recently updated codebases. For these old forks, you can set a "restart above" memory limit to automatically restart farming as they slowly memory leak over a day or so. Add 'RESTART_FARMER_IF_CONTAINER_MEMORY_EXCEEDS_GB = 3.0' to /root.chia/machinaris/config/api.cfg file, to restart above 3 GB for example.
  4. Harvesters use a small amount of memory (~hundred MB), BUT that scales with the number of plots they are harvesting. So a very large plot count will require more (GBs) of memory. You can split up your farm to many harvesters if you have more machines than RAM though.

When a blockchain is syncing (catching up to the current time), much more CPU is used. Once a blockchain is synced, the fullnode will use less CPU. Blockchains require some resources all the time however, this is the nature of crypto.

As Machinaris runs on Docker, one can apply the same CPU and memory controls that Docker makes available for any container. Be mindful that limiting CPU and memory too much make break farming/harvesting/plotting/etc.