Deploy Distributed MinIO with several nodes in Google Cloud - cniackz/public GitHub Wiki

Offical Documentation:

https://docs.min.io/minio/baremetal/installation/deploy-minio-distributed.html

Personal Documentation:

Created VMs using Randy's script:

$ ./benchmark_gcp_test.sh
Hello, what is your project id?:
1. daring-bit-354216
2. minio-benchmarking
2
PROJECT_ID is minio-benchmarking
what is your service account:
1. DoiT International: [email protected]
2. MinIO: [email protected]
2
SERVICE_ACCOUNT: [email protected]
How many nodes do you need to test?:
4
NUMBER_OF_NODES: 4
How many disks do you need to test with?:
4
NUMBER_OF_DISKS: 4
.
.
.

Install the MinIO Binary on Each Node

wget https://dl.min.io/server/minio/release/linux-amd64/archive/minio_0.0.20210116021944_amd64.deb -O minio.deb
sudo dpkg -i minio.deb

Create the user and group:

sudo groupadd -r minio-user
sudo useradd -M -r -g minio-user minio-user
sudo chown minio-user:minio-user /mnt/disks/disk1 /mnt/disks/disk1 /mnt/disks/disk1 /mnt/disks/disk1

Created file with Vars:

sudo touch /etc/default/minio

Content:

MINIO_VOLUMES="http://min-{1...4}/mnt/disks/disk{1...4}"
MINIO_OPTS="--console-address :9001"
MINIO_ROOT_USER=minioadmin
MINIO_ROOT_PASSWORD=minioadmin
MINIO_SERVER_URL="http://localhost:9000"

Started the service on each node and is working on each node:

sudo systemctl start minio.service
sudo systemctl status minio.service

cniackz@min-1:~$ sudo systemctl stop minio.service
cniackz@min-1:~$ sudo systemctl start minio.service
cniackz@min-1:~$ sudo systemctl status minio.service
● minio.service - MinIO
     Loaded: loaded (/etc/systemd/system/minio.service; disabled; vendor preset: enabled)
     Active: active (running) since Thu 2022-07-14 22:03:41 UTC; 14s ago
       Docs: https://docs.min.io
    Process: 1159 ExecStartPre=/bin/bash -c if [ -z "${MINIO_VOLUMES}" ]; then echo "Variable MINIO_VOLUMES not set in /etc/default>
   Main PID: 1160 (minio)
      Tasks: 11
     Memory: 96.3M
        CPU: 1.694s
     CGroup: /system.slice/minio.service
             └─1160 /usr/local/bin/minio server --console-address :9001 http://min-{1...4}/mnt/disks/disk{1...4}

Jul 14 22:03:49 min-1 minio[1160]: All MinIO sub-systems initialized successfully in 80.275151ms
Jul 14 22:03:49 min-1 minio[1160]: Waiting for all MinIO IAM sub-system to be initialized.. lock acquired
Jul 14 22:03:49 min-1 minio[1160]: MinIO Object Storage Server
Jul 14 22:03:49 min-1 minio[1160]: Copyright: 2015-2022 MinIO, Inc.
Jul 14 22:03:49 min-1 minio[1160]: License: GNU AGPLv3 <https://www.gnu.org/licenses/agpl-3.0.html>
Jul 14 22:03:49 min-1 minio[1160]: Version: RELEASE.2022-07-13T23-29-44Z (go1.18.4 linux/amd64)
Jul 14 22:03:49 min-1 minio[1160]: Status:         16 Online, 0 Offline.
Jul 14 22:03:49 min-1 minio[1160]: API: http://localhost:9000
Jul 14 22:03:49 min-1 minio[1160]: Console: http://10.0.0.22:9001 http://127.0.0.1:9001
Jul 14 22:03:49 min-1 minio[1160]: Documentation: https://docs.min.io

cniackz@min-2:~$ sudo systemctl status minio.service
● minio.service - MinIO
     Loaded: loaded (/etc/systemd/system/minio.service; disabled; vendor preset: enabled)
     Active: active (running) since Thu 2022-07-14 22:03:43 UTC; 15s ago
       Docs: https://docs.min.io
    Process: 1117 ExecStartPre=/bin/bash -c if [ -z "${MINIO_VOLUMES}" ]; then echo "Variable MINIO_VOLUMES not set in /etc/default>
   Main PID: 1118 (minio)
      Tasks: 10
     Memory: 167.9M
        CPU: 1.903s
     CGroup: /system.slice/minio.service
             └─1118 /usr/local/bin/minio server --console-address :9001 http://min-{1...4}/mnt/disks/disk{1...4}

Jul 14 22:03:50 min-2 minio[1118]: All MinIO sub-systems initialized successfully in 929.610456ms
Jul 14 22:03:50 min-2 minio[1118]: Waiting for all MinIO IAM sub-system to be initialized.. lock acquired
Jul 14 22:03:50 min-2 minio[1118]: MinIO Object Storage Server
Jul 14 22:03:50 min-2 minio[1118]: Copyright: 2015-2022 MinIO, Inc.
Jul 14 22:03:50 min-2 minio[1118]: License: GNU AGPLv3 <https://www.gnu.org/licenses/agpl-3.0.html>
Jul 14 22:03:50 min-2 minio[1118]: Version: RELEASE.2022-07-13T23-29-44Z (go1.18.4 linux/amd64)
Jul 14 22:03:50 min-2 minio[1118]: Status:         16 Online, 0 Offline.
Jul 14 22:03:50 min-2 minio[1118]: API: http://localhost:9000
Jul 14 22:03:50 min-2 minio[1118]: Console: http://10.0.0.23:9001 http://127.0.0.1:9001
Jul 14 22:03:50 min-2 minio[1118]: Documentation: https://docs.min.io

cniackz@min-3:~$ sudo systemctl status minio.service
● minio.service - MinIO
     Loaded: loaded (/etc/systemd/system/minio.service; disabled; vendor preset: enabled)
     Active: active (running) since Thu 2022-07-14 22:03:45 UTC; 16s ago
       Docs: https://docs.min.io
    Process: 1093 ExecStartPre=/bin/bash -c if [ -z "${MINIO_VOLUMES}" ]; then echo "Variable MINIO_VOLUMES not set in /etc/default>
   Main PID: 1094 (minio)
      Tasks: 11
     Memory: 152.0M
        CPU: 1.295s
     CGroup: /system.slice/minio.service
             └─1094 /usr/local/bin/minio server --console-address :9001 http://min-{1...4}/mnt/disks/disk{1...4}

Jul 14 22:03:50 min-3 minio[1094]: All MinIO sub-systems initialized successfully in 1.585750988s
Jul 14 22:03:51 min-3 minio[1094]: Waiting for all MinIO IAM sub-system to be initialized.. lock acquired
Jul 14 22:03:51 min-3 minio[1094]: MinIO Object Storage Server
Jul 14 22:03:51 min-3 minio[1094]: Copyright: 2015-2022 MinIO, Inc.
Jul 14 22:03:51 min-3 minio[1094]: License: GNU AGPLv3 <https://www.gnu.org/licenses/agpl-3.0.html>
Jul 14 22:03:51 min-3 minio[1094]: Version: RELEASE.2022-07-13T23-29-44Z (go1.18.4 linux/amd64)
Jul 14 22:03:51 min-3 minio[1094]: Status:         16 Online, 0 Offline.
Jul 14 22:03:51 min-3 minio[1094]: API: http://localhost:9000
Jul 14 22:03:51 min-3 minio[1094]: Console: http://10.0.0.24:9001 http://127.0.0.1:9001
Jul 14 22:03:51 min-3 minio[1094]: Documentation: https://docs.min.io

cniackz@min-4:~$ sudo systemctl status minio.service
● minio.service - MinIO
     Loaded: loaded (/etc/systemd/system/minio.service; disabled; vendor preset: enabled)
     Active: active (running) since Thu 2022-07-14 22:03:48 UTC; 15s ago
       Docs: https://docs.min.io
    Process: 1044 ExecStartPre=/bin/bash -c if [ -z "${MINIO_VOLUMES}" ]; then echo "Variable MINIO_VOLUMES not set in /etc/default>
   Main PID: 1045 (minio)
      Tasks: 11
     Memory: 93.1M
        CPU: 888ms
     CGroup: /system.slice/minio.service
             └─1045 /usr/local/bin/minio server --console-address :9001 http://min-{1...4}/mnt/disks/disk{1...4}

Jul 14 22:03:50 min-4 minio[1045]: All MinIO sub-systems initialized successfully in 500.982144ms
Jul 14 22:03:50 min-4 minio[1045]: Waiting for all MinIO IAM sub-system to be initialized.. lock acquired
Jul 14 22:03:50 min-4 minio[1045]: MinIO Object Storage Server
Jul 14 22:03:50 min-4 minio[1045]: Copyright: 2015-2022 MinIO, Inc.
Jul 14 22:03:50 min-4 minio[1045]: License: GNU AGPLv3 <https://www.gnu.org/licenses/agpl-3.0.html>
Jul 14 22:03:50 min-4 minio[1045]: Version: RELEASE.2022-07-13T23-29-44Z (go1.18.4 linux/amd64)
Jul 14 22:03:50 min-4 minio[1045]: Status:         16 Online, 0 Offline.
Jul 14 22:03:50 min-4 minio[1045]: API: http://localhost:9000
Jul 14 22:03:50 min-4 minio[1045]: Console: http://10.0.0.25:9001 http://127.0.0.1:9001
Jul 14 22:03:50 min-4 minio[1045]: Documentation: https://docs.min.io

Ran Speed Test after installing mc:

sudo apt-get install wget
sudo wget https://dl.min.io/server/minio/release/linux-amd64/minio
sudo wget https://dl.min.io/client/mc/release/linux-amd64/mc
sudo chmod +x minio
sudo chmod +x mc
sudo mv minio /usr/local/bin/minio
sudo mv mc /usr/local/bin/mc

cniackz@min-1:~$ sudo mc alias set myminio http://localhost:9000 minioadmin minioadmin
Added `myminio` successfully.
cniackz@min-1:~$ sudo mc support perf object myminio/

   	THROUGHPUT	IOPS     
PUT	256 MiB/s 	4 objs/s	
GET	256 MiB/s 	4 objs/s