U1.34 Ubuntu Quick Start (QS): NFS persistent storage on Kubernetes on premises - chempkovsky/CS2WPF-and-CS2XAMARIN GitHub Wiki

Before we start

read the article NFS Storage Classes
- we choose NFS subdir external provisioner
we continue to work with the Kubernetes cluster prepared in the article U1.33 Ubuntu Quick Start (QS): Kubernetes on premises and Docker and Kubespray.
- our hyper-v service is running under MS server 2016
  - MS server 2016 has three hard drives: C, D, E, i.e. a separate hard drive for each virtual machine of our Kubernetes cluster.

Click to show picture

Preparation step

the current step is required to avoid "Kubernetes NFS volume mount fail with exit status 32"
login into each machine
- u2004d01
- u2004d02
- u2004d03
run the following command

sudo apt-get install -y nfs-common

Configure NFS server storage folders under MS server 2016

on each drive of MS server 2016 we create the folder for NFS torage
- for drive c: C:\Hyper-V\nfsonc
- for drive d: D:\Hyper-V\nfsond
- for drive e: E:\Hyper-V\nfsone
run powershell
- run powershell commands
  - Note 1: "Get-WindowsFeature nfs": it shows if the feature is already installed
  - Note 2: Do not use Install-WindowsFeature FS-NFS-Service ...-command if the feature is already installed

Get-WindowsFeature *nfs*
Install-WindowsFeature FS-NFS-Service -IncludeAllSubFeature -IncludeManagementTools
New-NfsShare -Name 'nfsonc' -Path 'C:\Hyper-V\nfsonc' -EnableUnmappedAccess $True -Authentication sys
Grant-NfsSharePermission -Name 'nfsonc' -ClientName '192.168.100.2' -ClientType 'Host' -Permission 'readwrite' -AllowRootAccess:$true
Grant-NfsSharePermission -Name 'nfsonc' -ClientName '192.168.100.9' -ClientType 'Host' -Permission 'readwrite' -AllowRootAccess:$true
Grant-NfsSharePermission -Name 'nfsonc' -ClientName '192.168.100.13' -ClientType 'Host' -Permission 'readwrite' -AllowRootAccess:$true
Grant-NfsSharePermission -Name 'nfsonc' -ClientName '192.168.100.26' -ClientType 'Host' -Permission 'readwrite' -AllowRootAccess:$true
Get-NfsShare nfsonc
Get-NfsSharePermission -Name nfsonc

New-NfsShare -Name 'nfsond' -Path 'D:\Hyper-V\nfsond' -EnableUnmappedAccess $True -Authentication sys
Grant-NfsSharePermission -Name 'nfsond' -ClientName '192.168.100.2' -ClientType 'Host' -Permission 'readwrite' -AllowRootAccess:$true
Grant-NfsSharePermission -Name 'nfsond' -ClientName '192.168.100.9' -ClientType 'Host' -Permission 'readwrite' -AllowRootAccess:$true
Grant-NfsSharePermission -Name 'nfsond' -ClientName '192.168.100.13' -ClientType 'Host' -Permission 'readwrite' -AllowRootAccess:$true
Get-NfsShare nfsond
Get-NfsSharePermission -Name nfsond

New-NfsShare -Name 'nfsone' -Path 'E:\Hyper-V\nfsone' -EnableUnmappedAccess $True -Authentication sys
Grant-NfsSharePermission -Name 'nfsone' -ClientName '192.168.100.2' -ClientType 'Host' -Permission 'readwrite' -AllowRootAccess:$true
Grant-NfsSharePermission -Name 'nfsone' -ClientName '192.168.100.9' -ClientType 'Host' -Permission 'readwrite' -AllowRootAccess:$true
Grant-NfsSharePermission -Name 'nfsone' -ClientName '192.168.100.13' -ClientType 'Host' -Permission 'readwrite' -AllowRootAccess:$true
Get-NfsShare nfsone
Get-NfsSharePermission -Name nfsone

Note: for nfsonc we have added addtional permission -ClientName '192.168.100.26'. It's IP of UbuntuAnsible. We will use UbuntuAnsible for mount testing.

Click to show responce

PS C:\> New-NfsShare -Name 'nfsone' -Path 'E:\Hyper-V\nfsone' -EnableUnmappedAccess $True -Authentication sys

Name   Availability             Path
----   ------------             ----
nfsone Standard (not clustered) E:\Hyper-V\nfsone


PS C:\> Grant-NfsSharePermission -Name 'nfsone' -ClientName '192.168.100.2' -ClientType 'Host' -Permission 'readwrite' -AllowRootAccess:$true
PS C:\> Grant-NfsSharePermission -Name 'nfsone' -ClientName '192.168.100.9' -ClientType 'Host' -Permission 'readwrite' -AllowRootAccess:$true
PS C:\> Grant-NfsSharePermission -Name 'nfsone' -ClientName '192.168.100.13' -ClientType 'Host' -Permission 'readwrite' -AllowRootAccess:$true
PS C:\> Get-NfsShare nfsone

Name   Availability             Path
----   ------------             ----
nfsone Standard (not clustered) E:\Hyper-V\nfsone


PS C:\> Get-NfsSharePermission -Name nfsone

Name   ClientName     Permission  AllowRootAccess
----   ----------     ----------  ---------------
nfsone 192.168.100.2  READ, WRITE True
nfsone 192.168.100.9  READ, WRITE True
nfsone 192.168.100.13 READ, WRITE True
nfsone All Machines   DENY ACCESS False

Mount and Unmount Nfs-folder on UbuntuAnsible-machine

Note: IP address of Ms server 2016 = 192.168.100.8

Mount

sudo mkdir -p /nfs/test
sudo mount 192.168.100.8:/nfsonc /nfs/test
sudo nano /nfs/test/testlile.txt

Unmount

sudo umount /nfs/test

Install NFS subdir external provisioner

Note: IP address of Ms server 2016 = 192.168.100.8
read the article nfs-subdir-external-provisioner
login into u2004d01
run the commads (no sudo prefix for the second command)
- Note: second commad creates nfs-subdir-external-provisioner-folder

sudo apt install git
git clone https://github.com/kubernetes-sigs/nfs-subdir-external-provisioner
cd nfs-subdir-external-provisioner

Provisioner and storageClass for "nfsonc"-folder

run the commads (no sudo prefix for the command)
- inside the editor press ^O and rename the file deploy/rbacc.yaml (with letter "c")
- modify the file
  - replacing "namespace: default" with "namespace: nfsoncnmsp"
  - replace ALL "nfs-client-provisioner" with "nfsonc-client-provisioner"

nano deploy/rbac.yaml

Click to show deploy/rbacc.yaml

apiVersion: v1
kind: ServiceAccount
metadata:
  name: nfsonc-client-provisioner
  # replace with namespace where provisioner is deployed
  namespace: nfsoncnmsp
---
kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: nfsonc-client-provisioner-runner
rules:
  - apiGroups: [""]
    resources: ["nodes"]
    verbs: ["get", "list", "watch"]
  - apiGroups: [""]
    resources: ["persistentvolumes"]
    verbs: ["get", "list", "watch", "create", "delete"]
  - apiGroups: [""]
    resources: ["persistentvolumeclaims"]
    verbs: ["get", "list", "watch", "update"]
  - apiGroups: ["storage.k8s.io"]
    resources: ["storageclasses"]
    verbs: ["get", "list", "watch"]
  - apiGroups: [""]
    resources: ["events"]
    verbs: ["create", "update", "patch"]
---
kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: run-nfsonc-client-provisioner
subjects:
  - kind: ServiceAccount
    name: nfsonc-client-provisioner
    # replace with namespace where provisioner is deployed
    namespace: nfsoncnmsp
roleRef:
  kind: ClusterRole
  name: nfsonc-client-provisioner-runner
  apiGroup: rbac.authorization.k8s.io
---
kind: Role
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: leader-locking-nfsonc-client-provisioner
  # replace with namespace where provisioner is deployed
  namespace: nfsoncnmsp
rules:
  - apiGroups: [""]
    resources: ["endpoints"]
    verbs: ["get", "list", "watch", "create", "update", "patch"]
---
kind: RoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: leader-locking-nfsonc-client-provisioner
  # replace with namespace where provisioner is deployed
  namespace: nfsoncnmsp
subjects:
  - kind: ServiceAccount
    name: nfsonc-client-provisioner
    # replace with namespace where provisioner is deployed
    namespace: nfsoncnmsp
roleRef:
  kind: Role
  name: leader-locking-nfsonc-client-provisioner
  apiGroup: rbac.authorization.k8s.io

run the commads (no sudo prefix for the command)
- inside the editor press ^O and rename the file deploy/deploymentc.yaml (with letter "c")
- modify the file
  - replacing "namespace: default" with "namespace: nfsoncnmsp"
  - replace "nfs-client-provisioner" with "nfsonc-client-provisioner"
  - setting "env.PROVISIONER_NAME.value: nfsonc/nfs-subdir-external-provisioner"
  - setting NFS_SERVER.value: 192.168.100.8
  - setting NFS_PATH.value: /nfsonc
  - setting volumes.nfs.server: 192.1668.100.8
  - setting volumes.nfs.path: /nfsonc

nano deploy/deployment.yaml

Click to show deploy/deploymentc.yaml

apiVersion: apps/v1
kind: Deployment
metadata:
  name: nfsonc-client-provisioner
  labels:
    app: nfsonc-client-provisioner
  # replace with namespace where provisioner is deployed
  namespace: nfsoncnmsp
spec:
  replicas: 1
  strategy:
    type: Recreate
  selector:
    matchLabels:
      app: nfsonc-client-provisioner
  template:
    metadata:
      labels:
        app: nfsonc-client-provisioner
    spec:
      serviceAccountName: nfsonc-client-provisioner
      containers:
        - name: nfsonc-client-provisioner
          image: k8s.gcr.io/sig-storage/nfs-subdir-external-provisioner:v4.0.2
          volumeMounts:
            - name: nfs-client-root
              mountPath: /persistentvolumes
          env:
            - name: PROVISIONER_NAME
              value: nfsonc/nfs-subdir-external-provisioner
            - name: NFS_SERVER
              value: 192.168.100.8
            - name: NFS_PATH
              value: /nfsonc
      volumes:
        - name: nfs-client-root
          nfs:
            server: 192.168.100.8
            path: /nfsonc

run the commads (no sudo prefix for the command)
- inside the editor press ^O and rename the file deploy/classc.yaml (with letter "c")
- modify the file
  - setting "provisioner: nfsonc/nfs-subdir-external-provisioner"
  - replacing "name: managed-nfs-storage" with "name: managed-nfsonc-storage"

nano deploy/class.yaml

Click to show deploy/classc.yaml

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: managed-nfsonc-storage
provisioner: nfsonc/nfs-subdir-external-provisioner # or choose another name, must match deployment's env PROVISIONER_NAME'
parameters:
  archiveOnDelete: "false"

run the commads

sudo kubectl get namespaces
sudo kubectl create namespace nfsoncnmsp
NS=$(kubectl config get-contexts|grep -e "^\*" |awk '{print $5}')
sudo kubectl create -f deploy/rbacc.yaml
sudo kubectl create -f deploy/deploymentc.yaml
sudo kubectl create -f deploy/classc.yaml
sudo kubectl get storageClass --all-namespaces -o wide
sudo kubectl describe storageClass managed-nfsonc-storage

Provisioner and storageClass for "nfsond"-folder

run the commads (no sudo prefix for the command)
- inside the editor press ^O and rename the file deploy/rbacd.yaml (with letter "d")
- modify the file
  - replacing "namespace: default" with "namespace: nfsondnmsp"
  - replace ALL "nfs-client-provisioner" with "nfsond-client-provisioner"

nano deploy/rbac.yaml

Click to show deploy/rbacd.yaml

apiVersion: v1
kind: ServiceAccount
metadata:
  name: nfsond-client-provisioner
  # replace with namespace where provisioner is deployed
  namespace: nfsondnmsp
---
kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: nfsond-client-provisioner-runner
rules:
  - apiGroups: [""]
    resources: ["nodes"]
    verbs: ["get", "list", "watch"]
  - apiGroups: [""]
    resources: ["persistentvolumes"]
    verbs: ["get", "list", "watch", "create", "delete"]
  - apiGroups: [""]
    resources: ["persistentvolumeclaims"]
    verbs: ["get", "list", "watch", "update"]
  - apiGroups: ["storage.k8s.io"]
    resources: ["storageclasses"]
    verbs: ["get", "list", "watch"]
  - apiGroups: [""]
    resources: ["events"]
    verbs: ["create", "update", "patch"]
---
kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: run-nfsond-client-provisioner
subjects:
  - kind: ServiceAccount
    name: nfsond-client-provisioner
    # replace with namespace where provisioner is deployed
    namespace: nfsondnmsp
roleRef:
  kind: ClusterRole
  name: nfsond-client-provisioner-runner
  apiGroup: rbac.authorization.k8s.io
---
kind: Role
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: leader-locking-nfsond-client-provisioner
  # replace with namespace where provisioner is deployed
  namespace: nfsondnmsp
rules:
  - apiGroups: [""]
    resources: ["endpoints"]
    verbs: ["get", "list", "watch", "create", "update", "patch"]
---
kind: RoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: leader-locking-nfsond-client-provisioner
  # replace with namespace where provisioner is deployed
  namespace: nfsondnmsp
subjects:
  - kind: ServiceAccount
    name: nfsond-client-provisioner
    # replace with namespace where provisioner is deployed
    namespace: nfsondnmsp
roleRef:
  kind: Role
  name: leader-locking-nfsond-client-provisioner
  apiGroup: rbac.authorization.k8s.io

run the commads (no sudo prefix for the command)
- inside the editor press ^O and rename the file deploy/deploymentd.yaml (with letter "d")
- modify the file
  - replacing "namespace: default" with "namespace: nfsondnmsp"
  - replace "nfs-client-provisioner" with "nfsond-client-provisioner"
  - setting "env.PROVISIONER_NAME.value: nfsond/nfs-subdir-external-provisioner"
  - setting NFS_SERVER.value: 192.168.100.8
  - setting NFS_PATH.value: /nfsond
  - setting volumes.nfs.server: 192.1668.100.8
  - setting volumes.nfs.path: /nfsond

nano deploy/deployment.yaml

Click to show deploy/deploymentd.yaml

apiVersion: apps/v1
kind: Deployment
metadata:
  name: nfsond-client-provisioner
  labels:
    app: nfsond-client-provisioner
  # replace with namespace where provisioner is deployed
  namespace: nfsondnmsp
spec:
  replicas: 1
  strategy:
    type: Recreate
  selector:
    matchLabels:
      app: nfsond-client-provisioner
  template:
    metadata:
      labels:
        app: nfsond-client-provisioner
    spec:
      serviceAccountName: nfsond-client-provisioner
      containers:
        - name: nfsond-client-provisioner
          image: k8s.gcr.io/sig-storage/nfs-subdir-external-provisioner:v4.0.2
          volumeMounts:
            - name: nfs-client-root
              mountPath: /persistentvolumes
          env:
            - name: PROVISIONER_NAME
              value: nfsond/nfs-subdir-external-provisioner
            - name: NFS_SERVER
              value: 192.168.100.8
            - name: NFS_PATH
              value: /nfsond
      volumes:
        - name: nfs-client-root
          nfs:
            server: 192.168.100.8
            path: /nfsond

run the commads (no sudo prefix for the command)
- inside the editor press ^O and rename the file deploy/classe.yaml (with letter "e")
- modify the file
  - setting "provisioner: nfsond/nfs-subdir-external-provisioner"
  - replacing "name: managed-nfs-storage" with "name: managed-nfsond-storage"

nano deploy/class.yaml

Click to show deploy/classd.yaml

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: managed-nfsond-storage
provisioner: nfsond/nfs-subdir-external-provisioner # or choose another name, must match deployment's env PROVISIONER_NAME'
parameters:
  archiveOnDelete: "false"

run the commads

sudo kubectl get namespaces
sudo kubectl create namespace nfsondnmsp
NS=$(kubectl config get-contexts|grep -e "^\*" |awk '{print $5}')
sudo kubectl create -f deploy/rbacd.yaml
sudo kubectl create -f deploy/deploymentd.yaml
sudo kubectl create -f deploy/classd.yaml
sudo kubectl get storageClass --all-namespaces -o wide
sudo kubectl describe storageClass managed-nfsond-storage

Provisioner and storageClass for "nfsone"-folder

run the commads (no sudo prefix for the command)
- inside the editor press ^O and rename the file deploy/rbace.yaml (with letter "e")
- modify the file
  - replacing "namespace: default" with "namespace: nfsonenmsp"
  - replace ALL "nfs-client-provisioner" with "nfsone-client-provisioner"

nano deploy/rbac.yaml

Click to show deploy/rbace.yaml

apiVersion: v1
kind: ServiceAccount
metadata:
  name: nfsone-client-provisioner
  # replace with namespace where provisioner is deployed
  namespace: nfsonenmsp
---
kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: nfsone-client-provisioner-runner
rules:
  - apiGroups: [""]
    resources: ["nodes"]
    verbs: ["get", "list", "watch"]
  - apiGroups: [""]
    resources: ["persistentvolumes"]
    verbs: ["get", "list", "watch", "create", "delete"]
  - apiGroups: [""]
    resources: ["persistentvolumeclaims"]
    verbs: ["get", "list", "watch", "update"]
  - apiGroups: ["storage.k8s.io"]
    resources: ["storageclasses"]
    verbs: ["get", "list", "watch"]
  - apiGroups: [""]
    resources: ["events"]
    verbs: ["create", "update", "patch"]
---
kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: run-nfsone-client-provisioner
subjects:
  - kind: ServiceAccount
    name: nfsone-client-provisioner
    # replace with namespace where provisioner is deployed
    namespace: nfsonenmsp
roleRef:
  kind: ClusterRole
  name: nfsone-client-provisioner-runner
  apiGroup: rbac.authorization.k8s.io
---
kind: Role
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: leader-locking-nfsone-client-provisioner
  # replace with namespace where provisioner is deployed
  namespace: nfsonenmsp
rules:
  - apiGroups: [""]
    resources: ["endpoints"]
    verbs: ["get", "list", "watch", "create", "update", "patch"]
---
kind: RoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: leader-locking-nfsone-client-provisioner
  # replace with namespace where provisioner is deployed
  namespace: nfsonenmsp
subjects:
  - kind: ServiceAccount
    name: nfsone-client-provisioner
    # replace with namespace where provisioner is deployed
    namespace: nfsonenmsp
roleRef:
  kind: Role
  name: leader-locking-nfsone-client-provisioner
  apiGroup: rbac.authorization.k8s.io

run the commads (no sudo prefix for the command)
- inside the editor press ^O and rename the file deploy/deploymente.yaml (with letter "e")
- modify the file
  - replacing "namespace: default" with "namespace: nfsonenmsp"
  - replace "nfs-client-provisioner" with "nfsone-client-provisioner"
  - setting "env.PROVISIONER_NAME.value: nfsone/nfs-subdir-external-provisioner"
  - setting NFS_SERVER.value: 192.168.100.8
  - setting NFS_PATH.value: /nfsone
  - setting volumes.nfs.server: 192.1668.100.8
  - setting volumes.nfs.path: /nfsone

nano deploy/deployment.yaml

Click to show deploy/deploymente.yaml

apiVersion: apps/v1
kind: Deployment
metadata:
  name: nfsone-client-provisioner
  labels:
    app: nfsone-client-provisioner
  # replace with namespace where provisioner is deployed
  namespace: nfsonenmsp
spec:
  replicas: 1
  strategy:
    type: Recreate
  selector:
    matchLabels:
      app: nfsone-client-provisioner
  template:
    metadata:
      labels:
        app: nfsone-client-provisioner
    spec:
      serviceAccountName: nfsone-client-provisioner
      containers:
        - name: nfsone-client-provisioner
          image: k8s.gcr.io/sig-storage/nfs-subdir-external-provisioner:v4.0.2
          volumeMounts:
            - name: nfs-client-root
              mountPath: /persistentvolumes
          env:
            - name: PROVISIONER_NAME
              value: nfsone/nfs-subdir-external-provisioner
            - name: NFS_SERVER
              value: 192.168.100.8
            - name: NFS_PATH
              value: /nfsone
      volumes:
        - name: nfs-client-root
          nfs:
            server: 192.168.100.8
            path: /nfsone

run the commads (no sudo prefix for the command)
- inside the editor press ^O and rename the file deploy/classe.yaml (with letter "e")
- modify the file
  - setting "provisioner: nfsone/nfs-subdir-external-provisioner"
  - replacing "name: managed-nfs-storage" with "name: managed-nfsone-storage"

nano deploy/class.yaml

Click to show deploy/classe.yaml

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: managed-nfsone-storage
provisioner: nfsone/nfs-subdir-external-provisioner # or choose another name, must match deployment's env PROVISIONER_NAME'
parameters:
  archiveOnDelete: "false"

run the commads

sudo kubectl get namespaces
sudo kubectl create namespace nfsonenmsp
NS=$(kubectl config get-contexts|grep -e "^\*" |awk '{print $5}')
sudo kubectl create -f deploy/rbace.yaml
sudo kubectl create -f deploy/deploymente.yaml
sudo kubectl create -f deploy/classe.yaml
sudo kubectl get storageClass --all-namespaces -o wide
sudo kubectl describe storageClass managed-nfsone-storage