MinIO CPU Usage versus Time per PUT - cniackz/public GitHub Wiki

Objective:

To document an experiment that shows MinIO is getting slower and at the same time is consuming less CPU.

Hardware:

  • x86_64 Architecture (amd64)
  • Ubuntu 22.04
  • 16 GB RAM
  • 1 NVME Drive of 1 TB
$ uname -a
Linux asus 5.19.0-32-generic #33~22.04.1-Ubuntu SMP PREEMPT_DYNAMIC Mon Jan 30 17:03:34 UTC 2 x86_64 x86_64 x86_64 GNU/Linux

Pre-Steps:

  1. Created file of 100M for the PUT transactions:
dd if=/dev/null of=abc.bin obs=100M seek=1
                               |
                               |___ An object of 100M that takes around 1 sec per Object to be uploaded.

Steps:

  1. Execute MinIO:
# Window 1: The main process and the one we want to measure and compare.
cd ~/minio-versions/minio-RELEASE.XXXX-XX-XXTXX-XX-XXZ # Select the version to run.
export MINIO_ROOT_USER=minio
export MINIO_ROOT_PASSWORD=minio123
rm -rf /data/.minio.sys/; ./minio server /data --address :9000
                                           |
                                           |____ It is one nvme disk of 1TB
  1. The script to measure the CPU Usage and Time it takes for 1000 uploads of object above: ./script.sh <minio-process-id>
start=`date +%s`
top -b -p $1 > top-output.txt & # start measuring
FOO_PID=$! # Obtain the process id of the measuring process
for i in $(seq 1 1000);
do
  echo $i;
  mc cp abc.bin myminio/bucket/abc.bin; # It does the PUTs and stress the CPU
done
kill -9 $FOO_PID # kill the measurement
grep $1 top-output.txt | grep ccelis | grep minio > parsed-top-output.txt # Parsing the info
python3 measure.py # create an average of the CPU stress of all puts.
end=`date +%s`
echo Execution time was `expr $end - $start` seconds.
  1. In a 3rd window you can monitor the progress of the test:
### Window 3: Manual verify during the process.
grep 1720715 top-output.txt | grep ccelis | grep minio > parsed-top-output.txt; cat parsed-top-output.txt; python3 measure.py
        |                            |             |
        |                            |             |___ The command generating the process
        |                            |              
        |                            |___ The user executing the process
        |___ The process id

Python script to get the CPU average:

# CPU Usage:
number_of_lines=0
accumulator=float(0.0)
with open("parsed-top-output.txt") as file:
    for line in file:
        tmp_line = line.rstrip().split()
        accumulator=accumulator+float(tmp_line[-4])
        if float(tmp_line[-4]) > float(1):
            number_of_lines=number_of_lines+1
accumulator=accumulator/float(number_of_lines)
print(accumulator)

Where data is being saved:

https://docs.google.com/spreadsheets/d/10iWzO6hw5NZMP7MjRr3pb7fcRNe1z0PltYqO5Q-gz70/edit?usp=sharing

For minio binary, get those from repo and compile it locally:

  1. Downloaded zip
  2. unzip it
  3. compile it make build
  4. run it:
ccelis@asus:~/minio-versions/minio-RELEASE.2023-02-27T18-10-45Z$ ./minio server /data --address :9000
                                                                     |
                                                                     |___ Compiled from minio-RELEASE.2023-02-27T18-10-45Z

Results: