How to install directpv in added nodes - cniackz/public GitHub Wiki
Objective:
To show/provide a demo on how you can install DirectPV in new added nodes to an existing cluster.
Steps:
- Please read all documentation on how to upgrade DirectPV if not in the latest version:
- https://github.com/minio/directpv/blob/master/docs/upgrade.md
- https://asciinema.org/a/2Stv8ugsQg72rWOEWlLUVNWrV
- Once the documentation has been read, please proceed and upgrade accordingly.
- For this demo, initial version was:
minio@minio-k8s17:~$ kubectl directpv --version
directpv version v3.2.0
- We uninstall it:
kubectl directpv uninstall
Expected:
minio@minio-k8s17:~$ kubectl directpv uninstall
I0413 15:07:00.092622 2844108 deployment.go:302] 'direct.csi.min.io' controller deployment deleted
I0413 15:07:00.098556 2844108 daemonset.go:53] 'direct.csi.min.io' daemonset deleted
I0413 15:07:00.110232 2844108 service.go:47] 'direct.csi.min.io' service deleted
I0413 15:07:00.113472 2844108 validationrules.go:155] 'direct.csi.min.io' validation rules removed
I0413 15:07:01.184327 2844108 storageclass.go:60] 'direct.csi.min.io' storageclass deleted
I0413 15:07:01.967597 2844108 csidriver.go:158] 'direct.csi.min.io' csidriver deleted
I0413 15:07:01.974746 2844108 conversion_secret.go:42] 'direct.csi.min.io' conversion secrets deleted
W0413 15:07:01.987550 2844108 warnings.go:70] policy/v1beta1 PodSecurityPolicy is deprecated in v1.21+, unavailable in v1.25+
I0413 15:07:01.987604 2844108 psp.go:163] 'direct.csi.min.io' pod security policy removed
I0413 15:07:02.052471 2844108 rbac.go:377] 'direct.csi.min.io' rbac roles deleted
I0413 15:07:02.066227 2844108 ns.go:82] 'direct.csi.min.io' namespace deleted
-
We make sure no remaining pod is still present in the cluster.
-
Uninstalled the plugin:
minio@minio-k8s17:~$ kubectl krew uninstall directpv
Uninstalled plugin: directpv
- Then re-installed the plugin:
minio@minio-k8s17:~$ kubectl krew install directpv
Updated the local copy of plugin index.
Installing plugin: directpv
Installed plugin: directpv
\
| Use this plugin:
| kubectl directpv
| Documentation:
| https://github.com/minio/directpv
/
WARNING: You installed plugin "directpv" from the krew-index plugin repository.
These plugins are not audited for security by the Krew maintainers.
Run them at your own risk.
- Labeled the nodes to show how you can select the nodes where DirectPV will run:
kubectl label nodes minio-k8s17 directpv=yes
kubectl label nodes minio-k8s18 directpv=yes
kubectl label nodes minio-k8s19 directpv=yes
kubectl label nodes minio-k8s20 directpv=yes
- Finally the installation is made against those nodes:
minio@minio-k8s17:~$ kubectl directpv install --node-selector directpv=yes
███████████████████████████████████████████████████████████████████████████ 100%
┌──────────────────────────────────────┬──────────────────────────┐
│ NAME │ KIND │
├──────────────────────────────────────┼──────────────────────────┤
│ directpv │ Namespace │
│ directpv-min-io │ ServiceAccount │
│ directpv-min-io │ ClusterRole │
│ directpv-min-io │ ClusterRoleBinding │
│ directpv-min-io │ Role │
│ directpv-min-io │ RoleBinding │
│ directpv-min-io │ PodSecurityPolicy │
│ psp-directpv-min-io │ ClusterRoleBinding │
│ directpvdrives.directpv.min.io │ CustomResourceDefinition │
│ directpvvolumes.directpv.min.io │ CustomResourceDefinition │
│ directpvnodes.directpv.min.io │ CustomResourceDefinition │
│ directpvinitrequests.directpv.min.io │ CustomResourceDefinition │
│ directpv-min-io │ CSIDriver │
│ direct-csi-min-io │ CSIDriver │
│ directpv-min-io │ StorageClass │
│ direct-csi-min-io │ StorageClass │
│ node-server │ Daemonset │
│ legacy-node-server │ Daemonset │
│ controller │ Deployment │
│ legacy-controller │ Deployment │
└──────────────────────────────────────┴──────────────────────────┘
DirectPV installed successfully
- Version is now:
minio@minio-k8s17:~$ kubectl directpv --version
directpv version v4.0.3
- And DirectPV is running in selected nodes:
- We can see all drives:
minio@minio-k8s17:~$ kubectl directpv list drives
┌─────────────┬──────────┬─────────────────────┬────────┬────────┬─────────┬────────┐
│ NODE │ NAME │ MAKE │ SIZE │ FREE │ VOLUMES │ STATUS │
├─────────────┼──────────┼─────────────────────┼────────┼────────┼─────────┼────────┤
│ minio-k8s17 │ nvme1n1 │ INTEL SSDPE2NV153T8 │ 14 TiB │ 11 TiB │ 4 │ Ready │
│ minio-k8s17 │ nvme10n1 │ INTEL SSDPE2NV153T8 │ 14 TiB │ 12 TiB │ 3 │ Ready │
│ minio-k8s17 │ nvme11n1 │ INTEL SSDPE2NV153T8 │ 14 TiB │ 12 TiB │ 4 │ Ready │
│ minio-k8s17 │ nvme12n1 │ INTEL SSDPE2NV153T8 │ 14 TiB │ 12 TiB │ 3 │ Ready │
│ minio-k8s17 │ nvme2n1 │ INTEL SSDPE2NV153T8 │ 14 TiB │ 12 TiB │ 4 │ Ready │
│ minio-k8s17 │ nvme3n1 │ INTEL SSDPE2NV153T8 │ 14 TiB │ 12 TiB │ 4 │ Ready │
│ minio-k8s17 │ nvme4n1 │ INTEL SSDPE2NV153T8 │ 14 TiB │ 12 TiB │ 3 │ Ready │
│ minio-k8s17 │ nvme5n1 │ INTEL SSDPE2NV153T8 │ 14 TiB │ 12 TiB │ 3 │ Ready │
│ minio-k8s17 │ nvme6n1 │ INTEL SSDPE2NV153T8 │ 14 TiB │ 12 TiB │ 4 │ Ready │
│ minio-k8s17 │ nvme7n1 │ INTEL SSDPE2NV153T8 │ 14 TiB │ 12 TiB │ 3 │ Ready │
│ minio-k8s17 │ nvme8n1 │ INTEL SSDPE2NV153T8 │ 14 TiB │ 12 TiB │ 4 │ Ready │
│ minio-k8s17 │ nvme9n1 │ INTEL SSDPE2NV153T8 │ 14 TiB │ 12 TiB │ 3 │ Ready │
│ minio-k8s18 │ nvme3n1 │ INTEL SSDPE2NV153T8 │ 14 TiB │ 11 TiB │ 4 │ Ready │
│ minio-k8s18 │ nvme8n1 │ INTEL SSDPE2NV153T8 │ 14 TiB │ 11 TiB │ 4 │ Ready │
│ minio-k8s18 │ nvme10n1 │ INTEL SSDPE2NV153T8 │ 14 TiB │ 12 TiB │ 3 │ Ready │
│ minio-k8s18 │ nvme11n1 │ INTEL SSDPE2NV153T8 │ 14 TiB │ 12 TiB │ 3 │ Ready │
│ minio-k8s18 │ nvme12n1 │ INTEL SSDPE2NV153T8 │ 14 TiB │ 12 TiB │ 3 │ Ready │
│ minio-k8s18 │ nvme1n1 │ INTEL SSDPE2NV153T8 │ 14 TiB │ 12 TiB │ 3 │ Ready │
│ minio-k8s18 │ nvme2n1 │ INTEL SSDPE2NV153T8 │ 14 TiB │ 12 TiB │ 3 │ Ready │
│ minio-k8s18 │ nvme4n1 │ INTEL SSDPE2NV153T8 │ 14 TiB │ 12 TiB │ 3 │ Ready │
│ minio-k8s18 │ nvme5n1 │ INTEL SSDPE2NV153T8 │ 14 TiB │ 12 TiB │ 3 │ Ready │
│ minio-k8s18 │ nvme6n1 │ INTEL SSDPE2NV153T8 │ 14 TiB │ 12 TiB │ 3 │ Ready │
│ minio-k8s18 │ nvme7n1 │ INTEL SSDPE2NV153T8 │ 14 TiB │ 12 TiB │ 3 │ Ready │
│ minio-k8s18 │ nvme9n1 │ INTEL SSDPE2NV153T8 │ 14 TiB │ 12 TiB │ 3 │ Ready │
│ minio-k8s19 │ nvme10n1 │ INTEL SSDPE2NV153T8 │ 14 TiB │ 12 TiB │ 3 │ Ready │
│ minio-k8s19 │ nvme11n1 │ INTEL SSDPE2NV153T8 │ 14 TiB │ 12 TiB │ 4 │ Ready │
│ minio-k8s19 │ nvme12n1 │ INTEL SSDPE2NV153T8 │ 14 TiB │ 12 TiB │ 4 │ Ready │
│ minio-k8s19 │ nvme1n1 │ INTEL SSDPE2NV153T8 │ 14 TiB │ 12 TiB │ 3 │ Ready │
│ minio-k8s19 │ nvme2n1 │ INTEL SSDPE2NV153T8 │ 14 TiB │ 12 TiB │ 3 │ Ready │
│ minio-k8s19 │ nvme3n1 │ INTEL SSDPE2NV153T8 │ 14 TiB │ 12 TiB │ 4 │ Ready │
│ minio-k8s19 │ nvme4n1 │ INTEL SSDPE2NV153T8 │ 14 TiB │ 12 TiB │ 4 │ Ready │
│ minio-k8s19 │ nvme5n1 │ INTEL SSDPE2NV153T8 │ 14 TiB │ 12 TiB │ 3 │ Ready │
│ minio-k8s19 │ nvme6n1 │ INTEL SSDPE2NV153T8 │ 14 TiB │ 12 TiB │ 5 │ Ready │
│ minio-k8s19 │ nvme7n1 │ INTEL SSDPE2NV153T8 │ 14 TiB │ 12 TiB │ 3 │ Ready │
│ minio-k8s19 │ nvme8n1 │ INTEL SSDPE2NV153T8 │ 14 TiB │ 12 TiB │ 5 │ Ready │
│ minio-k8s19 │ nvme9n1 │ INTEL SSDPE2NV153T8 │ 14 TiB │ 12 TiB │ 3 │ Ready │
│ minio-k8s20 │ nvme10n1 │ INTEL SSDPE2NV153T8 │ 14 TiB │ 12 TiB │ 3 │ Ready │
│ minio-k8s20 │ nvme11n1 │ INTEL SSDPE2NV153T8 │ 14 TiB │ 12 TiB │ 4 │ Ready │
│ minio-k8s20 │ nvme12n1 │ INTEL SSDPE2NV153T8 │ 14 TiB │ 12 TiB │ 3 │ Ready │
│ minio-k8s20 │ nvme1n1 │ INTEL SSDPE2NV153T8 │ 14 TiB │ 12 TiB │ 3 │ Ready │
│ minio-k8s20 │ nvme2n1 │ INTEL SSDPE2NV153T8 │ 14 TiB │ 12 TiB │ 2 │ Ready │
│ minio-k8s20 │ nvme3n1 │ INTEL SSDPE2NV153T8 │ 14 TiB │ 12 TiB │ 3 │ Ready │
│ minio-k8s20 │ nvme4n1 │ INTEL SSDPE2NV153T8 │ 14 TiB │ 12 TiB │ 3 │ Ready │
│ minio-k8s20 │ nvme5n1 │ INTEL SSDPE2NV153T8 │ 14 TiB │ 12 TiB │ 4 │ Ready │
│ minio-k8s20 │ nvme6n1 │ INTEL SSDPE2NV153T8 │ 14 TiB │ 12 TiB │ 4 │ Ready │
│ minio-k8s20 │ nvme7n1 │ INTEL SSDPE2NV153T8 │ 14 TiB │ 12 TiB │ 2 │ Ready │
│ minio-k8s20 │ nvme8n1 │ INTEL SSDPE2NV153T8 │ 14 TiB │ 12 TiB │ 3 │ Ready │
│ minio-k8s20 │ nvme9n1 │ INTEL SSDPE2NV153T8 │ 14 TiB │ 12 TiB │ 3 │ Ready │
└─────────────┴──────────┴─────────────────────┴────────┴────────┴─────────┴────────┘
- And since this is existing DirectPV cluster, no initialization is longer needed:
minio@minio-k8s17:~$ kubectl directpv discover
Discovered node 'minio-k8s17' ✔
Discovered node 'minio-k8s18' ✔
Discovered node 'minio-k8s19' ✔
Discovered node 'minio-k8s20' ✔
No drives are available to initialize
- Test DirectPV can provision Persistent Volumes:
- Use this claim as example and modify it accordingly, call it
claim.yaml:
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: please-remove-me-cesar
namespace: ns-3
annotations:
volume.beta.kubernetes.io/storage-provisioner: direct-csi-min-io
volume.kubernetes.io/selected-node: minio-k8s18
finalizers:
- kubernetes.io/pvc-protection
spec:
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 5Gi
storageClassName: directpv-min-io
volumeMode: Filesystem
- Apply it:
kubectl apply -f claim.yaml
- Check it gets
Bound:
$ k get pv pvc-49ad809a-9521-41a1-a9f4-20fb0b007194
NAME CAPACITY ACCESS MODES RECLAIM POLICY STATUS CLAIM STORAGECLASS REASON AGE
pvc-49ad809a-9521-41a1-a9f4-20fb0b007194 5Gi RWO Delete Bound ns-3/please-remove-me-cesar directpv-min-io 94s
Cesars-MacBook-Pro:~ cniackz$