Increasing the number of executors on Jenkins Master (built‐in) Node - TerrenceMcGuinness-NOAA/global-workflow GitHub Wiki

Increasing the number of executors on the Jenkins master node from 16 to 32 will significantly increase the load on the system. To ensure that your Jenkins master node can handle this increased load, you need to consider several factors, including CPU, memory, disk I/O, and network bandwidth.

System Requirements for Increasing Executors

Here are some general guidelines for system requirements when increasing the number of executors:

  1. CPU:

    • Each executor can be thought of as a separate thread that will consume CPU resources.
    • For 32 executors, you should have a multi-core CPU with at least 16-32 cores to ensure that the system can handle the concurrent load. More cores are better, especially if your jobs are CPU-intensive.

  2. Memory (RAM):

    • Jenkins itself requires a significant amount of memory, and each executor will also consume memory.
    • For 32 executors, you should have at least 64 GB of RAM. If your jobs are memory-intensive, you may need more. A good rule of thumb is to allocate 2 GB of RAM per executor, plus additional memory for Jenkins and the operating system.

  3. Disk I/O:

    • Jenkins jobs often involve reading and writing large amounts of data to disk. Fast disk I/O is crucial to avoid bottlenecks.
    • Use SSDs for the Jenkins home directory and workspace directories to ensure fast read/write speeds.
    • Ensure you have sufficient disk space to handle the build artifacts and logs generated by 32 concurrent jobs.

  4. Network Bandwidth:

    • If your jobs involve downloading dependencies or uploading artifacts, ensure that your network bandwidth can handle the increased load.
    • A high-speed network connection (e.g., 1 Gbps or higher) is recommended.

  5. Operating System:

    • Ensure that your operating system is optimized for handling multiple concurrent processes. Use a 64-bit OS to take full advantage of the available memory.

Example System Configuration

Here is an example of a system configuration that could handle 32 executors:

  • CPU: 16-32 cores (e.g., dual 16-core processors)
  • Memory: 64-128 GB of RAM
  • Disk: SSDs with sufficient capacity (e.g., 1 TB or more)
  • Network: 1 Gbps or higher network connection
  • Operating System: 64-bit Linux distribution (e.g., Ubuntu, CentOS)

Monitoring and Tuning

When increasing the number of executors, it's important to monitor the system's performance and make adjustments as needed:

  1. Monitor CPU Usage: Use tools like top, htop, or vmstat to monitor CPU usage and ensure that the system is not becoming CPU-bound.

  2. Monitor Memory Usage: Use tools like free, vmstat, or top to monitor memory usage and ensure that the system is not running out of memory.

  3. Monitor Disk I/O: Use tools like iostat or iotop to monitor disk I/O and ensure that the system is not becoming I/O-bound.

  4. Monitor Network Usage: Use tools like iftop or nload to monitor network usage and ensure that the network is not becoming a bottleneck.

  5. Jenkins Monitoring: Use Jenkins monitoring plugins (e.g., Monitoring Plugin, Performance Plugin) to monitor the performance of Jenkins itself.

By carefully considering these factors and monitoring the system's performance, you can determine an appropriate limit for the number of executors and ensure that your Jenkins master node can handle the increased load effectively.