decommission - cniackz/public GitHub Wiki
Objective:
A manual way to test decommission when fails are observed in our CI/CD Pipeline
Steps
- Create cluster
createcluster
- install operator
installoperator
- install tenant lite
installtenant
- increase from 1 pool to 2 pools
- Wait couple of minutes until is ready:
- Put some data
mc alias set myminio https://127.0.0.1:30082 minio minio123 --insecure
mc mb myminio/celis --insecure
echo "a" > a.txt
mc cp a.txt myminio/celis --insecure
mc ls myminio/celis --insecure
- Get status:
mc admin decommission status myminio --insecure
Expected:
$ mc admin decommission status myminio --insecure
┌─────┬───────────────────────────────────────────────────────────────────────────────────────┬────────────────────────┬────────┐
│ ID │ Pools │ Raw Drives Usage │ Status │
│ 1st │ https://myminio-pool-0-{0...3}.myminio-hl.tenant-lite.svc.cluster.local/export{0...1} │ 16.2% (total: 332 GiB) │ Active │
│ 2nd │ https://myminio-pool-1-{0...3}.myminio-hl.tenant-lite.svc.cluster.local/export{0...1} │ 16.2% (total: 332 GiB) │ Active │
└─────┴───────────────────────────────────────────────────────────────────────────────────────┴────────────────────────┴────────┘
- Start Decommission:
mc admin decommission start myminio/ https://myminio-pool-0-{0...3}.myminio-hl.tenant-lite.svc.cluster.local/export{0...1} --insecure
Expected:
$ mc admin decommission start myminio/ https://myminio-pool-0-{0...3}.myminio-hl.tenant-lite.svc.cluster.local/export{0...1} --insecure
Decommission started successfully for `https://myminio-pool-0-{0...3}.myminio-hl.tenant-lite.svc.cluster.local/export{0...1}`.
- Monitor until complete:
$ mc admin decommission status myminio/ --insecure
┌─────┬───────────────────────────────────────────────────────────────────────────────────────┬────────────────────────┬──────────┐
│ ID │ Pools │ Raw Drives Usage │ Status │
│ 1st │ https://myminio-pool-0-{0...3}.myminio-hl.tenant-lite.svc.cluster.local/export{0...1} │ 16.2% (total: 332 GiB) │ Complete │
│ 2nd │ https://myminio-pool-1-{0...3}.myminio-hl.tenant-lite.svc.cluster.local/export{0...1} │ 16.2% (total: 332 GiB) │ Active │
└─────┴───────────────────────────────────────────────────────────────────────────────────────┴────────────────────────┴──────────┘
- Remove decommissioned pool from tenant spec
k edit tenant myminio -n tenant-lite
Cesars-MacBook-Pro:~ cniackz$ k edit tenant storage-lite -n tenant-lite
tenant.minio.min.io/storage-lite edited
- Operator log:
I0130 15:18:40.193678 1 event.go:298] Event(v1.ObjectReference{Kind:"Tenant", Namespace:"tenant-lite", Name:"myminio", UID:"58a44125-aff4-4b93-8b7f-95367bc4fceb", APIVersion:"minio.min.io/v2", ResourceVersion:"2509", FieldPath:""}): type: 'Normal' reason: 'PoolRemoved' Tenant pool myminio-pool-0 removed
- Status:
$ mc admin decommission status myminio --insecure
┌─────┬───────────────────────────────────────────────────────────────────────────────────────┬────────────────────────┬────────┐
│ ID │ Pools │ Raw Drives Usage │ Status │
│ 2nd │ https://myminio-pool-1-{0...3}.myminio-hl.tenant-lite.svc.cluster.local/export{0...1} │ 14.9% (total: 332 GiB) │ Active │
└─────┴───────────────────────────────────────────────────────────────────────────────────────┴────────────────────────┴────────┘
- Pods from pool 0 are gone, pods from pool 1 are present:
$ k get pods -n tenant-lite
NAME READY STATUS RESTARTS AGE
myminio-pool-1-0 2/2 Running 0 9m40s
myminio-pool-1-1 2/2 Running 0 9m43s
myminio-pool-1-2 2/2 Running 0 9m46s
myminio-pool-1-3 2/2 Running 0 10m
- Check data is still present after decommission:
$ mc ls myminio/celis --insecure
[2024-04-11 16:23:22 EDT] 2B STANDARD a.txt
Additional Notes:
- Decommission worked in 4.5.8
- Decommission worked in 5.0.11