U1.36 Ubuntu Quick Start (QS): StorageOS or Ondat persistent storage on Kubernetes on premises. Out of date documentation and Errors. - chempkovsky/CS2WPF-and-CS2XAMARIN GitHub Wiki
- philosophy
- read the article How to Solve Kubernetes’ Persistent Storage Challenges. It references four solutions:
- Longhorn
- OpenEBS
- Rook/
- StorageOS
- read the article Stateful Apps in Kubernetes are a big deal
- read the article How to Solve Kubernetes’ Persistent Storage Challenges. It references four solutions:
- practices
- read the article Create Kubernetes Persistent Volume 4/10/2021
- we continue to work with the Kubernetes cluster prepared in the article U1.33 Ubuntu Quick Start (QS): Kubernetes on premises and Docker and Kubespray.
- our hyper-v service is running under MS server 2016
- etcd settings of the installed Kubernetes cluster
sudo nano /etc/etcd.env
Click to show /etc/etcd.env
# Environment file for etcd v3.5.0
ETCD_DATA_DIR=/var/lib/etcd
ETCD_ADVERTISE_CLIENT_URLS=https://192.168.100.2:2379
ETCD_INITIAL_ADVERTISE_PEER_URLS=https://192.168.100.2:2380
ETCD_INITIAL_CLUSTER_STATE=existing
ETCD_METRICS=basic
ETCD_LISTEN_CLIENT_URLS=https://192.168.100.2:2379,https://127.0.0.1:2379
ETCD_ELECTION_TIMEOUT=5000
ETCD_HEARTBEAT_INTERVAL=250
ETCD_INITIAL_CLUSTER_TOKEN=k8s_etcd
ETCD_LISTEN_PEER_URLS=https://192.168.100.2:2380
ETCD_NAME=etcd1
ETCD_PROXY=off
ETCD_INITIAL_CLUSTER=etcd1=https://192.168.100.2:2380,etcd2=https://192.168.100.9:2380,etcd3=https://192.168.100.13:2380
ETCD_AUTO_COMPACTION_RETENTION=8
ETCD_SNAPSHOT_COUNT=10000
# Flannel need etcd v2 API
ETCD_ENABLE_V2=true
# TLS settings
ETCD_TRUSTED_CA_FILE=/etc/ssl/etcd/ssl/ca.pem
ETCD_CERT_FILE=/etc/ssl/etcd/ssl/member-u2004d01.pem
ETCD_KEY_FILE=/etc/ssl/etcd/ssl/member-u2004d01-key.pem
ETCD_CLIENT_CERT_AUTH=true
ETCD_PEER_TRUSTED_CA_FILE=/etc/ssl/etcd/ssl/ca.pem
ETCD_PEER_CERT_FILE=/etc/ssl/etcd/ssl/member-u2004d01.pem
ETCD_PEER_KEY_FILE=/etc/ssl/etcd/ssl/member-u2004d01-key.pem
ETCD_PEER_CLIENT_CERT_AUTH=True
# CLI settings
ETCDCTL_ENDPOINTS=https://127.0.0.1:2379
ETCDCTL_CACERT=/etc/ssl/etcd/ssl/ca.pem
ETCDCTL_KEY=/etc/ssl/etcd/ssl/admin-u2004d01-key.pem
ETCDCTL_CERT=/etc/ssl/etcd/ssl/admin-u2004d01.pem
- getting etcd member list according to the etcd CLI settings above
etcdctl --endpoints https://127.0.0.1:2379 --cacert="/etc/ssl/etcd/ssl/ca.pem" --cert="/etc/ssl/etcd/ssl/admin-u2004d01.pem" --key="/etc/ssl/etcd/ssl/admin-u2004d01-key.pem" member list
Click to show etcd member list
root@u2004d01:/# etcdctl --endpoints https://127.0.0.1:2379 --cacert="/etc/ssl/etcd/ssl/ca.pem" --cert="/etc/ssl/etcd/ssl/admin-u2004d01.pem" --key="/etc/ssl/etcd/ssl/admin-u2004d01-key.pem" member list
e2e2d0613c5501cc, started, etcd3, https://192.168.100.13:2380, https://192.168.100.13:2379, false
f7365f07d3953ea7, started, etcd1, https://192.168.100.2:2380, https://192.168.100.2:2379, false
f81d7ef9707e683f, started, etcd2, https://192.168.100.9:2380, https://192.168.100.9:2379, false
-
Note: As part of our setup, etcd runs outside Kubernetes. So, Ephemeral pods within Kubernetes (Testing) will not work.
- If you don't try, you don't know
-
Step 1
- read the article Ubuntu Package Installation
- for each vitual machine u2004d01, u2004d02, u2004d03
- run the commands
sudo apt -y update
sudo apt -y install linux-modules-extra-$(uname -r)
Click to show the responce
yury@u2004d01:~$ sudo apt -y install linux-modules-extra-$(uname -r)
Reading package lists... Done
Building dependency tree
Reading state information... Done
linux-modules-extra-5.11.0-41-generic is already the newest version (5.11.0-41.45~20.04.1).
linux-modules-extra-5.11.0-41-generic set to manually installed.
The following packages were automatically installed and are no longer required:
apt-clone archdetect-deb cryptsetup-bin dctrl-tools dmeventd dmraid dpkg-repack
gir1.2-timezonemap-1.0 gir1.2-xkl-1.0 kpartx kpartx-boot libaio1 libdebian-installer4
libdevmapper-event1.02.1 libdmraid1.0.0.rc16 libfprint-2-tod1 libllvm9 liblvm2cmd2.03
libtimezonemap-data libtimezonemap1 linux-modules-extra-5.4.0-26-generic lvm2 python3-icu
python3-pam rdate thin-provisioning-tools
Use 'sudo apt autoremove' to remove them.
0 upgraded, 0 newly installed, 0 to remove and 30 not upgraded.
-
Step 2
- read the article etcd: Installation Step by Step
- for the vitual machine u2004d01
- run the commands
- we use apiVersion: rbac.authorization.k8s.io/v1 instead of apiVersion: rbac.authorization.k8s.io/v1beta1
- we use MY_POD_NAMESPACE and MY_POD_NAME as is
- run the commands
sudo -i
export NAMESPACE=storageos-etcd
kubectl create namespace $NAMESPACE
kubectl -n $NAMESPACE create -f-<<END
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
name: etcd-operator
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: etcd-operator
subjects:
- kind: ServiceAccount
name: default
namespace: $NAMESPACE
END
kubectl -n $NAMESPACE create -f-<<END
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
name: etcd-operator
rules:
- apiGroups:
- etcd.database.coreos.com
resources:
- etcdclusters
- etcdbackups
- etcdrestores
verbs:
- "*"
- apiGroups:
- apiextensions.k8s.io
resources:
- customresourcedefinitions
verbs:
- "*"
- apiGroups:
- ""
resources:
- pods
- services
- endpoints
- persistentvolumeclaims
- events
verbs:
- "*"
- apiGroups:
- apps
resources:
- deployments
verbs:
- "*"
# The following permissions can be removed if not using S3 backup and TLS
- apiGroups:
- ""
resources:
- secrets
verbs:
- get
END
kubectl -n $NAMESPACE create -f - <<END
apiVersion: apps/v1
kind: Deployment
metadata:
name: etcd-operator
spec:
selector:
matchLabels:
app: etcd-operator
replicas: 1
template:
metadata:
labels:
app: etcd-operator
spec:
containers:
- name: etcd-operator
image: quay.io/coreos/etcd-operator:v0.9.4
command:
- etcd-operator
env:
- name: MY_POD_NAMESPACE
valueFrom:
fieldRef:
fieldPath: metadata.namespace
- name: MY_POD_NAME
valueFrom:
fieldRef:
fieldPath: metadata.name
END
kubectl -n $NAMESPACE get pod -lapp=etcd-operator
Click to show the responce
namespace/storageos-etcd created
clusterrolebinding.rbac.authorization.k8s.io/etcd-operator created
clusterrole.rbac.authorization.k8s.io/etcd-operator created
deployment.apps/etcd-operator created
root@u2004d01:~# kubectl -n $NAMESPACE get pod -lapp=etcd-operator
NAME READY STATUS RESTARTS AGE
etcd-operator-64689467f6-smbvf 1/1 Running 0 32s
-
Step 3
-
before command under "6. Create the EtcdCluster resource" according to the article
-
we run
-
kubectl get pods -n storageos-etcd
kubectl logs -n storageos-etcd etcd-operator-64689467f6-smbvf
Click to show the responce
root@u2004d01:~# kubectl get pods -n storageos-etcd
NAME READY STATUS RESTARTS AGE
etcd-operator-64689467f6-smbvf 1/1 Running 0 11m
root@u2004d01:~# kubectl logs -n storageos-etcd etcd-operator-64689467f6-smbvf
time="2021-12-02T09:23:14Z" level=info msg="etcd-operator Version: 0.9.4"
time="2021-12-02T09:23:14Z" level=info msg="Git SHA: c8a1c64"
time="2021-12-02T09:23:14Z" level=info msg="Go Version: go1.11.5"
time="2021-12-02T09:23:14Z" level=info msg="Go OS/Arch: linux/amd64"
E1202 09:23:14.999642 1 event.go:259] Could not construct reference to: '&v1.Endpoints{TypeMeta:v1.TypeMeta{Kind:"", APIVersion:""}, ObjectMeta:v1.ObjectMeta{Name:"etcd-operator", GenerateName:"", Namespace:"storageos-etcd", SelfLink:"", UID:"8ceef176-6c5a-4a30-bb87-35266e3e4086", ResourceVersion:"7076", Generation:0, CreationTimestamp:v1.Time{Time:time.Time{wall:0x0, ext:63774033794, loc:(*time.Location)(0x20d4640)}}, DeletionTimestamp:(*v1.Time)(nil), DeletionGracePeriodSeconds:(*int64)(nil), Labels:map[string]string(nil), Annotations:map[string]string{"control-plane.alpha.kubernetes.io/leader":"{\"holderIdentity\":\"etcd-operator-64689467f6-smbvf\",\"leaseDurationSeconds\":15,\"acquireTime\":\"2021-12-02T09:23:14Z\",\"renewTime\":\"2021-12-02T09:23:14Z\",\"leaderTransitions\":0}"}, OwnerReferences:[]v1.OwnerReference(nil), Initializers:(*v1.Initializers)(nil), Finalizers:[]string(nil), ClusterName:""}, Subsets:[]v1.EndpointSubset(nil)}' due to: 'selfLink was empty, can't make reference'. Will not report event: 'Normal' 'LeaderElection' 'etcd-operator-64689467f6-smbvf became leader'
time="2021-12-02T09:23:15Z" level=error msg="initialization failed: fail to init CRD: failed to create CRD: the server could not find the requested resource" pkg=controller
time="2021-12-02T09:23:15Z" level=info msg="retry in 30s..." pkg=controller
time="2021-12-02T09:23:45Z" level=error msg="initialization failed: fail to init CRD: failed to create CRD: the server could not find the requested resource" pkg=controller
time="2021-12-02T09:23:45Z" level=info msg="retry in 30s..." pkg=controller
time="2021-12-02T09:24:15Z" level=error msg="initialization failed: fail to init CRD: failed to create CRD: the server could not find the requested resource" pkg=controller
time="2021-12-02T09:24:15Z" level=info msg="retry in 30s..." pkg=controller
...
-
Step 4
- the next command throws an error. (this is "6. Create the EtcdCluster resource")
kubectl -n $NAMESPACE create -f - <<END
apiVersion: "etcd.database.coreos.com/v1beta2"
kind: "EtcdCluster"
metadata:
name: "storageos-etcd"
spec:
size: 3
version: "3.4.7"
pod:
etcdEnv:
- name: ETCD_QUOTA_BACKEND_BYTES
value: "2147483648" # 2 GB
- name: ETCD_AUTO_COMPACTION_RETENTION
value: "1000" # Keep 1000 revisions (default)
- name: ETCD_AUTO_COMPACTION_MODE
value: "revision" # Set the revision mode
resources:
requests:
cpu: 200m
memory: 300Mi
securityContext:
runAsNonRoot: true
runAsUser: 9000
fsGroup: 9000
tolerations:
- operator: "Exists"
affinity:
podAntiAffinity:
preferredDuringSchedulingIgnoredDuringExecution:
- weight: 100
podAffinityTerm:
labelSelector:
matchExpressions:
- key: etcd_cluster
operator: In
values:
- storageos-etcd
topologyKey: kubernetes.io/hostname
END
- responce
error: unable to recognize "STDIN": no matches for kind "EtcdCluster" in version "etcd.database.coreos.com/v1beta2"
- read the article Production - Etcd on External Virtual Machines
- Since we already have three ** ectd ** installed, we will not install three additional nodes. We will use existing ones instead.
- they recommend to use their distribution and their version of etcd:
export ETCD_VERSION="3.4.9"
export CLIENT_PORT="2379"
export PEERS_PORT="2380"
- On the other hand, the etcd version we installed with Kubespary is 3.5
- Any way
- If you don't try, you don't know
- read the article Install Ondat on Kubernetes 1.21
- login with sudo privilege
- run the command
kubectl create -f https://github.com/storageos/cluster-operator/releases/download/v2.4.4/storageos-operator.yaml
- got five errors:
Click to show the responce
unable to recognize "https://github.com/storageos/cluster-operator/releases/download/v2.4.4/storageos-operator.yaml": no matches for kind "CustomResourceDefinition" in version "apiextensions.k8s.io/v1beta1"
unable to recognize "https://github.com/storageos/cluster-operator/releases/download/v2.4.4/storageos-operator.yaml": no matches for kind "CustomResourceDefinition" in version "apiextensions.k8s.io/v1beta1"
unable to recognize "https://github.com/storageos/cluster-operator/releases/download/v2.4.4/storageos-operator.yaml": no matches for kind "CustomResourceDefinition" in version "apiextensions.k8s.io/v1beta1"
unable to recognize "https://github.com/storageos/cluster-operator/releases/download/v2.4.4/storageos-operator.yaml": no matches for kind "CustomResourceDefinition" in version "apiextensions.k8s.io/v1beta1"
unable to recognize "https://github.com/storageos/cluster-operator/releases/download/v2.4.4/storageos-operator.yaml": no matches for kind "ClusterRoleBinding" in version "rbac.authorization.k8s.io/v1beta1"
- run the command
kubectl delete -f https://github.com/storageos/cluster-operator/releases/download/v2.4.4/storageos-operator.yaml
- download and modify storageos-operator.yaml
- replace ".k8s.io/v1beta1" with ".k8s.io/v1" five times
Click to show new version of storageos-operator.yaml
---
apiVersion: apiextensions.k8s.io/v1
kind: CustomResourceDefinition
metadata:
name: storageosclusters.storageos.com
spec:
additionalPrinterColumns:
- JSONPath: .status.ready
description: Ready status of the storageos nodes.
name: ready
type: string
- JSONPath: .status.phase
description: Status of the whole cluster.
name: status
type: string
- JSONPath: .metadata.creationTimestamp
name: age
type: date
group: storageos.com
names:
kind: StorageOSCluster
listKind: StorageOSClusterList
plural: storageosclusters
shortNames:
- stos
singular: storageoscluster
scope: Namespaced
subresources:
status: {}
validation:
openAPIV3Schema:
description: StorageOSCluster is the Schema for the storageosclusters API
properties:
apiVersion:
description: 'APIVersion defines the versioned schema of this representation
of an object. Servers should convert recognized schemas to the latest
internal value, and may reject unrecognized values. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#resources'
type: string
kind:
description: 'Kind is a string value representing the REST resource this
object represents. Servers may infer this from the endpoint the client
submits requests to. Cannot be updated. In CamelCase. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds'
type: string
metadata:
type: object
spec:
description: StorageOSClusterSpec defines the desired state of StorageOSCluster
properties:
csi:
description: CSI defines the configurations for CSI.
properties:
deploymentStrategy:
type: string
deviceDir:
type: string
driverRegisterationMode:
type: string
driverRequiresAttachment:
type: string
enable:
type: boolean
enableControllerExpandCreds:
type: boolean
enableControllerPublishCreds:
type: boolean
enableNodePublishCreds:
type: boolean
enableProvisionCreds:
type: boolean
endpoint:
type: string
kubeletDir:
type: string
kubeletRegistrationPath:
type: string
pluginDir:
type: string
registrarSocketDir:
type: string
registrationDir:
type: string
version:
type: string
type: object
debug:
description: Debug is to set debug mode of the cluster.
type: boolean
disableFencing:
description: "Disable Pod Fencing. With StatefulSets, Pods are only
re-scheduled if the Pod has been marked as killed. In practice this
means that failover of a StatefulSet pod is a manual operation. \n
By enabling Pod Fencing and setting the `storageos.com/fenced=true`
label on a Pod, StorageOS will enable automated Pod failover (by killing
the application Pod on the failed node) if the following conditions
exist: \n - Pod fencing has not been explicitly disabled. - StorageOS
has determined that the node the Pod is running on is offline. StorageOS
uses Gossip and TCP checks and will retry for 30 seconds. At this
point all volumes on the failed node are marked offline (irrespective
of whether fencing is enabled) and volume failover starts. - The
Pod has the label `storageos.com/fenced=true` set. - The Pod has at
least one StorageOS volume attached. - Each StorageOS volume has at
least 1 healthy replica. \n When Pod Fencing is disabled, StorageOS
will not perform any interaction with Kubernetes when it detects that
a node has gone offline. Additionally, the Kubernetes permissions
required for Fencing will not be added to the StorageOS role."
type: boolean
disableScheduler:
description: Disable StorageOS scheduler extender.
type: boolean
disableTCMU:
description: "Disable TCMU can be set to true to disable the TCMU storage
driver. This is required when there are multiple storage systems
running on the same node and you wish to avoid conflicts. Only one
TCMU-based storage system can run on a node at a time. \n Disabling
TCMU will degrade performance."
type: boolean
disableTelemetry:
description: Disable Telemetry.
type: boolean
forceTCMU:
description: "Force TCMU can be set to true to ensure that TCMU is enabled
or cause StorageOS to abort startup. \n At startup, StorageOS will
automatically fallback to non-TCMU mode if another TCMU-based storage
system is running on the node. Since non-TCMU will degrade performance,
this may not always be desired."
type: boolean
images:
description: Images defines the various container images used in the
cluster.
properties:
apiManagerContainer:
type: string
csiClusterDriverRegistrarContainer:
type: string
csiExternalAttacherContainer:
type: string
csiExternalProvisionerContainer:
type: string
csiExternalResizerContainer:
type: string
csiLivenessProbeContainer:
type: string
csiNodeDriverRegistrarContainer:
type: string
hyperkubeContainer:
type: string
initContainer:
type: string
kubeSchedulerContainer:
type: string
nfsContainer:
type: string
nodeContainer:
type: string
type: object
ingress:
description: Ingress defines the ingress configurations used in the
cluster.
properties:
annotations:
additionalProperties:
type: string
type: object
enable:
type: boolean
hostname:
type: string
tls:
type: boolean
type: object
join:
description: Join is the join token used for service discovery.
type: string
k8sDistro:
description: "K8sDistro is the name of the Kubernetes distribution where
the operator is being deployed. It should be in the format: `name[-1.0]`,
where the version is optional and should only be appended if known.
\ Suitable names include: `openshift`, `rancher`, `aks`, `gke`, `eks`,
or the deployment method if using upstream directly, e.g `minishift`
or `kubeadm`. \n Setting k8sDistro is optional, and will be used to
simplify cluster configuration by setting appropriate defaults for
the distribution. The distribution information will also be included
in the product telemetry (if enabled), to help focus development efforts."
type: string
kvBackend:
description: KVBackend defines the key-value store backend used in the
cluster.
properties:
address:
type: string
backend:
type: string
type: object
namespace:
description: Namespace is the kubernetes Namespace where storageos resources
are provisioned.
type: string
nodeSelectorTerms:
description: NodeSelectorTerms is to set the placement of storageos
pods using node affinity requiredDuringSchedulingIgnoredDuringExecution.
items:
description: A null or empty node selector term matches no objects.
The requirements of them are ANDed. The TopologySelectorTerm type
implements a subset of the NodeSelectorTerm.
properties:
matchExpressions:
description: A list of node selector requirements by node's labels.
items:
description: A node selector requirement is a selector that
contains values, a key, and an operator that relates the key
and values.
properties:
key:
description: The label key that the selector applies to.
type: string
operator:
description: Represents a key's relationship to a set of
values. Valid operators are In, NotIn, Exists, DoesNotExist.
Gt, and Lt.
type: string
values:
description: An array of string values. If the operator
is In or NotIn, the values array must be non-empty. If
the operator is Exists or DoesNotExist, the values array
must be empty. If the operator is Gt or Lt, the values
array must have a single element, which will be interpreted
as an integer. This array is replaced during a strategic
merge patch.
items:
type: string
type: array
required:
- key
- operator
type: object
type: array
matchFields:
description: A list of node selector requirements by node's fields.
items:
description: A node selector requirement is a selector that
contains values, a key, and an operator that relates the key
and values.
properties:
key:
description: The label key that the selector applies to.
type: string
operator:
description: Represents a key's relationship to a set of
values. Valid operators are In, NotIn, Exists, DoesNotExist.
Gt, and Lt.
type: string
values:
description: An array of string values. If the operator
is In or NotIn, the values array must be non-empty. If
the operator is Exists or DoesNotExist, the values array
must be empty. If the operator is Gt or Lt, the values
array must have a single element, which will be interpreted
as an integer. This array is replaced during a strategic
merge patch.
items:
type: string
type: array
required:
- key
- operator
type: object
type: array
type: object
type: array
pause:
description: Pause is to pause the operator for the cluster.
type: boolean
resources:
description: Resources is to set the resource requirements of the storageos
containers.
properties:
limits:
additionalProperties:
anyOf:
- type: integer
- type: string
pattern: ^(\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))))?$
x-kubernetes-int-or-string: true
description: 'Limits describes the maximum amount of compute resources
allowed. More info: https://kubernetes.io/docs/concepts/configuration/manage-compute-resources-container/'
type: object
requests:
additionalProperties:
anyOf:
- type: integer
- type: string
pattern: ^(\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))))?$
x-kubernetes-int-or-string: true
description: 'Requests describes the minimum amount of compute resources
required. If Requests is omitted for a container, it defaults
to Limits if that is explicitly specified, otherwise to an implementation-defined
value. More info: https://kubernetes.io/docs/concepts/configuration/manage-compute-resources-container/'
type: object
type: object
secretRefName:
description: SecretRefName is the name of the secret object that contains
all the sensitive cluster configurations.
type: string
secretRefNamespace:
description: SecretRefNamespace is the namespace of the secret reference.
type: string
service:
description: Service is the Service configuration for the cluster nodes.
properties:
annotations:
additionalProperties:
type: string
type: object
externalPort:
type: integer
internalPort:
type: integer
name:
type: string
type:
type: string
required:
- name
- type
type: object
sharedDir:
description: 'SharedDir is the shared directory to be used when the
kubelet is running in a container. Typically: "/var/lib/kubelet/plugins/kubernetes.io~storageos".
If not set, defaults will be used.'
type: string
storageClassName:
description: StorageClassName is the name of default StorageClass created
for StorageOS volumes.
type: string
tlsEtcdSecretRefName:
description: TLSEtcdSecretRefName is the name of the secret object that
contains the etcd TLS certs. This secret is shared with etcd, therefore
it's not part of the main storageos secret.
type: string
tlsEtcdSecretRefNamespace:
description: TLSEtcdSecretRefNamespace is the namespace of the etcd
TLS secret object.
type: string
tolerations:
description: Tolerations is to set the placement of storageos pods using
pod toleration.
items:
description: The pod this Toleration is attached to tolerates any
taint that matches the triple <key,value,effect> using the matching
operator <operator>.
properties:
effect:
description: Effect indicates the taint effect to match. Empty
means match all taint effects. When specified, allowed values
are NoSchedule, PreferNoSchedule and NoExecute.
type: string
key:
description: Key is the taint key that the toleration applies
to. Empty means match all taint keys. If the key is empty, operator
must be Exists; this combination means to match all values and
all keys.
type: string
operator:
description: Operator represents a key's relationship to the value.
Valid operators are Exists and Equal. Defaults to Equal. Exists
is equivalent to wildcard for value, so that a pod can tolerate
all taints of a particular category.
type: string
tolerationSeconds:
description: TolerationSeconds represents the period of time the
toleration (which must be of effect NoExecute, otherwise this
field is ignored) tolerates the taint. By default, it is not
set, which means tolerate the taint forever (do not evict).
Zero and negative values will be treated as 0 (evict immediately)
by the system.
format: int64
type: integer
value:
description: Value is the taint value the toleration matches to.
If the operator is Exists, the value should be empty, otherwise
just a regular string.
type: string
type: object
type: array
required:
- secretRefName
- secretRefNamespace
type: object
status:
description: StorageOSClusterStatus defines the observed state of StorageOSCluster
properties:
members:
description: MembersStatus stores the status details of cluster member
nodes.
properties:
ready:
description: Ready are the storageos cluster members that are ready
to serve requests. The member names are the same as the node IPs.
items:
type: string
type: array
unready:
description: Unready are the storageos cluster nodes not ready to
serve requests.
items:
type: string
type: array
type: object
nodeHealthStatus:
additionalProperties:
description: NodeHealth contains health status of a node.
properties:
directfsInitiator:
type: string
director:
type: string
kv:
type: string
kvWrite:
type: string
nats:
type: string
presentation:
type: string
rdb:
type: string
type: object
type: object
nodes:
items:
type: string
type: array
phase:
description: ClusterPhase is the phase of the storageos cluster at a
given point in time.
type: string
ready:
type: string
type: object
type: object
version: v1
versions:
- name: v1
served: true
storage: true
---
apiVersion: apiextensions.k8s.io/v1
kind: CustomResourceDefinition
metadata:
name: storageosupgrades.storageos.com
spec:
group: storageos.com
names:
kind: StorageOSUpgrade
listKind: StorageOSUpgradeList
plural: storageosupgrades
singular: storageosupgrade
scope: Namespaced
subresources:
status: {}
validation:
openAPIV3Schema:
description: StorageOSUpgrade is the Schema for the storageosupgrades API
properties:
apiVersion:
description: 'APIVersion defines the versioned schema of this representation
of an object. Servers should convert recognized schemas to the latest
internal value, and may reject unrecognized values. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#resources'
type: string
kind:
description: 'Kind is a string value representing the REST resource this
object represents. Servers may infer this from the endpoint the client
submits requests to. Cannot be updated. In CamelCase. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds'
type: string
metadata:
type: object
spec:
description: StorageOSUpgradeSpec defines the desired state of StorageOSUpgrade
properties:
newImage:
description: NewImage is the new StorageOS node container image.
type: string
required:
- newImage
type: object
status:
description: StorageOSUpgradeStatus defines the observed state of StorageOSUpgrade
properties:
completed:
description: Completed is the status of upgrade process.
type: boolean
type: object
type: object
version: v1
versions:
- name: v1
served: true
storage: true
---
apiVersion: apiextensions.k8s.io/v1
kind: CustomResourceDefinition
metadata:
name: jobs.storageos.com
spec:
group: storageos.com
names:
kind: Job
listKind: JobList
plural: jobs
singular: job
scope: Namespaced
subresources:
status: {}
validation:
openAPIV3Schema:
description: Job is the Schema for the jobs API
properties:
apiVersion:
description: 'APIVersion defines the versioned schema of this representation
of an object. Servers should convert recognized schemas to the latest
internal value, and may reject unrecognized values. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#resources'
type: string
kind:
description: 'Kind is a string value representing the REST resource this
object represents. Servers may infer this from the endpoint the client
submits requests to. Cannot be updated. In CamelCase. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds'
type: string
metadata:
type: object
spec:
description: JobSpec defines the desired state of Job
properties:
args:
description: Args is an array of strings passed as an argument to the
job container.
items:
type: string
type: array
completionWord:
description: CompletionWord is the word that's looked for in the pod
logs to find out if a DaemonSet Pod has completed its task.
type: string
hostPath:
description: HostPath is the path in the host that's mounted into a
job container.
type: string
image:
description: Image is the container image to run as the job.
type: string
labelSelector:
description: LabelSelector is the label selector for the job Pods.
type: string
mountPath:
description: MountPath is the path in the job container where a volume
is mounted.
type: string
nodeSelectorTerms:
description: NodeSelectorTerms is the set of placement of the job pods
using node affinity requiredDuringSchedulingIgnoredDuringExecution.
items:
description: A null or empty node selector term matches no objects.
The requirements of them are ANDed. The TopologySelectorTerm type
implements a subset of the NodeSelectorTerm.
properties:
matchExpressions:
description: A list of node selector requirements by node's labels.
items:
description: A node selector requirement is a selector that
contains values, a key, and an operator that relates the key
and values.
properties:
key:
description: The label key that the selector applies to.
type: string
operator:
description: Represents a key's relationship to a set of
values. Valid operators are In, NotIn, Exists, DoesNotExist.
Gt, and Lt.
type: string
values:
description: An array of string values. If the operator
is In or NotIn, the values array must be non-empty. If
the operator is Exists or DoesNotExist, the values array
must be empty. If the operator is Gt or Lt, the values
array must have a single element, which will be interpreted
as an integer. This array is replaced during a strategic
merge patch.
items:
type: string
type: array
required:
- key
- operator
type: object
type: array
matchFields:
description: A list of node selector requirements by node's fields.
items:
description: A node selector requirement is a selector that
contains values, a key, and an operator that relates the key
and values.
properties:
key:
description: The label key that the selector applies to.
type: string
operator:
description: Represents a key's relationship to a set of
values. Valid operators are In, NotIn, Exists, DoesNotExist.
Gt, and Lt.
type: string
values:
description: An array of string values. If the operator
is In or NotIn, the values array must be non-empty. If
the operator is Exists or DoesNotExist, the values array
must be empty. If the operator is Gt or Lt, the values
array must have a single element, which will be interpreted
as an integer. This array is replaced during a strategic
merge patch.
items:
type: string
type: array
required:
- key
- operator
type: object
type: array
type: object
type: array
tolerations:
description: Tolerations is to set the placement of storageos pods using
pod toleration.
items:
description: The pod this Toleration is attached to tolerates any
taint that matches the triple <key,value,effect> using the matching
operator <operator>.
properties:
effect:
description: Effect indicates the taint effect to match. Empty
means match all taint effects. When specified, allowed values
are NoSchedule, PreferNoSchedule and NoExecute.
type: string
key:
description: Key is the taint key that the toleration applies
to. Empty means match all taint keys. If the key is empty, operator
must be Exists; this combination means to match all values and
all keys.
type: string
operator:
description: Operator represents a key's relationship to the value.
Valid operators are Exists and Equal. Defaults to Equal. Exists
is equivalent to wildcard for value, so that a pod can tolerate
all taints of a particular category.
type: string
tolerationSeconds:
description: TolerationSeconds represents the period of time the
toleration (which must be of effect NoExecute, otherwise this
field is ignored) tolerates the taint. By default, it is not
set, which means tolerate the taint forever (do not evict).
Zero and negative values will be treated as 0 (evict immediately)
by the system.
format: int64
type: integer
value:
description: Value is the taint value the toleration matches to.
If the operator is Exists, the value should be empty, otherwise
just a regular string.
type: string
type: object
type: array
required:
- args
- completionWord
- hostPath
- image
- mountPath
type: object
status:
description: JobStatus defines the observed state of Job
properties:
completed:
description: Completed indicates the complete status of job.
type: boolean
type: object
type: object
version: v1
versions:
- name: v1
served: true
storage: true
---
apiVersion: apiextensions.k8s.io/v1
kind: CustomResourceDefinition
metadata:
name: nfsservers.storageos.com
spec:
additionalPrinterColumns:
- JSONPath: .status.phase
description: Status of the NFS server.
name: status
type: string
- JSONPath: .spec.resources.requests.storage
description: Capacity of the NFS server.
name: capacity
type: string
- JSONPath: .status.remoteTarget
description: Remote target address of the NFS server.
name: target
type: string
- JSONPath: .status.accessModes
description: Access modes supported by the NFS server.
name: access modes
type: string
- JSONPath: .spec.storageClassName
description: StorageClass used for creating the NFS volume.
name: storageclass
type: string
- JSONPath: .metadata.creationTimestamp
name: age
type: date
group: storageos.com
names:
kind: NFSServer
listKind: NFSServerList
plural: nfsservers
shortNames:
- nfsserver
singular: nfsserver
scope: Namespaced
subresources:
status: {}
validation:
openAPIV3Schema:
description: NFSServer is the Schema for the nfsservers API
properties:
apiVersion:
description: 'APIVersion defines the versioned schema of this representation
of an object. Servers should convert recognized schemas to the latest
internal value, and may reject unrecognized values. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#resources'
type: string
kind:
description: 'Kind is a string value representing the REST resource this
object represents. Servers may infer this from the endpoint the client
submits requests to. Cannot be updated. In CamelCase. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds'
type: string
metadata:
type: object
spec:
description: NFSServerSpec defines the desired state of NFSServer
properties:
annotations:
additionalProperties:
type: string
description: The annotations-related configuration to add/set on each
Pod related object.
type: object
export:
description: The parameters to configure the NFS export
properties:
name:
description: Name of the export
type: string
persistentVolumeClaim:
description: PVC from which the NFS daemon gets storage for sharing
properties:
claimName:
description: 'ClaimName is the name of a PersistentVolumeClaim
in the same namespace as the pod using this volume. More info:
https://kubernetes.io/docs/concepts/storage/persistent-volumes#persistentvolumeclaims'
type: string
readOnly:
description: Will force the ReadOnly setting in VolumeMounts.
Default false.
type: boolean
required:
- claimName
type: object
server:
description: The NFS server configuration
properties:
accessMode:
description: Reading and Writing permissions on the export Valid
values are "ReadOnly", "ReadWrite" and "none"
type: string
squash:
description: This prevents the root users connected remotely
from having root privileges Valid values are "none", "rootid",
"root", and "all"
type: string
type: object
type: object
mountOptions:
description: PV mount options. Not validated - mount of the PVs will
simply fail if one is invalid.
items:
type: string
type: array
nfsContainer:
description: NFSContainer is the container image to use for the NFS
server.
type: string
persistentVolumeClaim:
description: PersistentVolumeClaim is the PVC source of the PVC to be
used with the NFS Server. If not specified, a new PVC is provisioned
and used.
properties:
claimName:
description: 'ClaimName is the name of a PersistentVolumeClaim in
the same namespace as the pod using this volume. More info: https://kubernetes.io/docs/concepts/storage/persistent-volumes#persistentvolumeclaims'
type: string
readOnly:
description: Will force the ReadOnly setting in VolumeMounts. Default
false.
type: boolean
required:
- claimName
type: object
persistentVolumeReclaimPolicy:
description: Reclamation policy for the persistent volume shared to
the user's pod.
type: string
resources:
description: Resources represents the minimum resources required
properties:
limits:
additionalProperties:
anyOf:
- type: integer
- type: string
pattern: ^(\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))))?$
x-kubernetes-int-or-string: true
description: 'Limits describes the maximum amount of compute resources
allowed. More info: https://kubernetes.io/docs/concepts/configuration/manage-compute-resources-container/'
type: object
requests:
additionalProperties:
anyOf:
- type: integer
- type: string
pattern: ^(\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))))?$
x-kubernetes-int-or-string: true
description: 'Requests describes the minimum amount of compute resources
required. If Requests is omitted for a container, it defaults
to Limits if that is explicitly specified, otherwise to an implementation-defined
value. More info: https://kubernetes.io/docs/concepts/configuration/manage-compute-resources-container/'
type: object
type: object
storageClassName:
description: StorageClassName is the name of the StorageClass used by
the NFS volume.
type: string
tolerations:
description: Tolerations is to set the placement of NFS server pods
using pod toleration.
items:
description: The pod this Toleration is attached to tolerates any
taint that matches the triple <key,value,effect> using the matching
operator <operator>.
properties:
effect:
description: Effect indicates the taint effect to match. Empty
means match all taint effects. When specified, allowed values
are NoSchedule, PreferNoSchedule and NoExecute.
type: string
key:
description: Key is the taint key that the toleration applies
to. Empty means match all taint keys. If the key is empty, operator
must be Exists; this combination means to match all values and
all keys.
type: string
operator:
description: Operator represents a key's relationship to the value.
Valid operators are Exists and Equal. Defaults to Equal. Exists
is equivalent to wildcard for value, so that a pod can tolerate
all taints of a particular category.
type: string
tolerationSeconds:
description: TolerationSeconds represents the period of time the
toleration (which must be of effect NoExecute, otherwise this
field is ignored) tolerates the taint. By default, it is not
set, which means tolerate the taint forever (do not evict).
Zero and negative values will be treated as 0 (evict immediately)
by the system.
format: int64
type: integer
value:
description: Value is the taint value the toleration matches to.
If the operator is Exists, the value should be empty, otherwise
just a regular string.
type: string
type: object
type: array
type: object
status:
description: NFSServerStatus defines the observed state of NFSServer
properties:
accessModes:
description: AccessModes is the access modes supported by the NFS server.
type: string
phase:
description: "Phase is a simple, high-level summary of where the NFS
Server is in its lifecycle. Phase will be set to Ready when the NFS
Server is ready for use. It is intended to be similar to the PodStatus
Phase described at: https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.14/#podstatus-v1-core
\n There are five possible phase values: - Pending: The NFS Server
has been accepted by the Kubernetes system, but one or more of
the components has not been created. This includes time before
being scheduled as well as time spent downloading images over
the network, which could take a while. - Running: The NFS Server
has been bound to a node, and all of the dependencies have been
created. - Succeeded: All NFS Server dependencies have terminated
in success, and will not be restarted. - Failed: All NFS Server
dependencies in the pod have terminated, and at least one container
has terminated in failure. The container either exited with non-zero
status or was terminated by the system. - Unknown: For some reason
the state of the NFS Server could not be obtained, typically due
to an error in communicating with the host of the pod."
type: string
remoteTarget:
description: RemoteTarget is the connection string that clients can
use to access the shared filesystem.
type: string
type: object
type: object
version: v1
versions:
- name: v1
served: true
storage: true
---
apiVersion: v1
kind: Namespace
metadata:
name: storageos-operator
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
creationTimestamp: null
name: storageos-operator
labels:
app.kubernetes.io/name: storageos-operator
app.kubernetes.io/instance: storageos-cluster-operator
app.kubernetes.io/component: operator
app.kubernetes.io/part-of: storageos
rules:
- apiGroups:
- storageos.com
resources:
- storageosclusters
- storageosclusters/status
- storageosupgrades
- storageosupgrades/status
- jobs
- jobs/status
- nfsservers
- nfsservers/status
verbs:
- '*'
- apiGroups:
- apps
resources:
- statefulsets
- daemonsets
- deployments
- replicasets
verbs:
- '*'
- apiGroups:
- ""
resources:
- nodes
verbs:
- list
- watch
- get
- update
- create
- patch
- apiGroups:
- ""
resources:
- pods
verbs:
- list
- watch
- get
- update
- patch
- delete
- create
- apiGroups:
- ""
resources:
- events
- namespaces
- serviceaccounts
- secrets
- services
- services/status
- services/finalizers
- persistentvolumeclaims
- persistentvolumeclaims/status
- persistentvolumes
- configmaps
- configmaps/status
- replicationcontrollers
- pods/binding
- pods/status
- endpoints
- endpoints/status
verbs:
- create
- patch
- get
- list
- delete
- watch
- update
- apiGroups:
- rbac.authorization.k8s.io
resources:
- roles
- rolebindings
- clusterroles
- clusterrolebindings
verbs:
- create
- delete
- apiGroups:
- storage.k8s.io
resources:
- storageclasses
- volumeattachments
- volumeattachments/status
- csinodeinfos
- csinodes
- csidrivers
verbs:
- create
- delete
- watch
- list
- get
- update
- patch
- apiGroups:
- apiextensions.k8s.io
resources:
- customresourcedefinitions
verbs:
- create
- delete
- apiGroups:
- csi.storage.k8s.io
resources:
- csidrivers
verbs:
- create
- delete
- apiGroups:
- policy
resources:
- poddisruptionbudgets
verbs:
- list
- watch
- apiGroups:
- security.openshift.io
resourceNames:
- privileged
resources:
- securitycontextconstraints
verbs:
- create
- delete
- update
- get
- use
- apiGroups:
- admissionregistration.k8s.io
resources:
- mutatingwebhookconfigurations
verbs:
- '*'
- apiGroups:
- monitoring.coreos.com
resources:
- servicemonitors
verbs:
- '*'
- apiGroups:
- apps
resources:
- deployments/finalizers
resourceNames:
- storageos-cluster-operator
verbs:
- update
- apiGroups:
- events.k8s.io
resources:
- events
verbs:
- create
- patch
- apiGroups:
- coordination.k8s.io
resources:
- leases
verbs:
- get
- create
- update
---
apiVersion: v1
kind: ServiceAccount
metadata:
name: storageoscluster-operator-sa
namespace: storageos-operator
labels:
app: storageos
app.kubernetes.io/name: storageos-operator
app.kubernetes.io/instance: storageos-cluster-operator
app.kubernetes.io/component: operator
app.kubernetes.io/part-of: storageos
---
kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
name: storageoscluster-operator-rolebinding
labels:
app.kubernetes.io/name: storageos-operator
app.kubernetes.io/instance: storageos-cluster-operator
app.kubernetes.io/component: operator
app.kubernetes.io/part-of: storageos
subjects:
- kind: ServiceAccount
name: storageoscluster-operator-sa
namespace: storageos-operator
roleRef:
kind: ClusterRole
name: storageos-operator
apiGroup: rbac.authorization.k8s.io
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: storageos-cluster-operator
namespace: storageos-operator
labels:
app.kubernetes.io/name: storageos-operator
app.kubernetes.io/instance: storageos-cluster-operator
app.kubernetes.io/component: operator
app.kubernetes.io/part-of: storageos
spec:
replicas: 1
selector:
matchLabels:
name: storageos-cluster-operator
template:
metadata:
labels:
name: storageos-cluster-operator
spec:
serviceAccountName: storageoscluster-operator-sa
containers:
- name: storageos-cluster-operator
image: storageos/cluster-operator:v2.4.4
ports:
- containerPort: 8383
name: metrics
- containerPort: 8686
name: operatormetrics
command:
- cluster-operator
imagePullPolicy: IfNotPresent
env:
- name: WATCH_NAMESPACE
valueFrom:
fieldRef:
fieldPath: metadata.namespace
- name: POD_NAME
valueFrom:
fieldRef:
fieldPath: metadata.name
- name: POD_NAMESPACE
valueFrom:
fieldRef:
fieldPath: metadata.namespace
- name: OPERATOR_NAME
value: "storageos-cluster-operator"
- run the command
nano storageos-operator.yaml
- insert new copy of storageos-operator.yaml and save the file
- run the command with local copy of storageos-operator.yaml file
kubectl create -f storageos-operator.yaml --validate=false
Click to show the responce
yury@u2004d01:~$ sudo kubectl create -f storageos-operator.yaml --validate=false
namespace/storageos-operator created
clusterrole.rbac.authorization.k8s.io/storageos-operator created
serviceaccount/storageoscluster-operator-sa created
clusterrolebinding.rbac.authorization.k8s.io/storageoscluster-operator-rolebinding created
deployment.apps/storageos-cluster-operator created
Error from server (Invalid): error when creating "storageos-operator.yaml": CustomResourceDefinition.apiextensions.k8s.io "storageosclusters.storageos.com" is invalid: spec.versions[0].schema.openAPIV3Schema: Required value: schemas are required
Error from server (Invalid): error when creating "storageos-operator.yaml": CustomResourceDefinition.apiextensions.k8s.io "storageosupgrades.storageos.com" is invalid: spec.versions[0].schema.openAPIV3Schema: Required value: schemas are required
Error from server (Invalid): error when creating "storageos-operator.yaml": CustomResourceDefinition.apiextensions.k8s.io "jobs.storageos.com" is invalid: spec.versions[0].schema.openAPIV3Schema: Required value: schemas are required
Error from server (Invalid): error when creating "storageos-operator.yaml": CustomResourceDefinition.apiextensions.k8s.io "nfsservers.storageos.com" is invalid: spec.versions[0].schema.openAPIV3Schema: Required value: schemas are required
- run the command
sudo kubectl -n storageos-operator get pod
Click to show the responce
yury@u2004d01:~$ sudo kubectl -n storageos-operator get pod
NAME READY STATUS RESTARTS AGE
storageos-cluster-operator-57797fcdb-lcr9m 0/1 CrashLoopBackOff 5 (28s ago) 3m37s
- run the command
sudo kubectl logs -n storageos-operator storageos-cluster-operator-57797fcdb-lcr9m
Click to show the responce
yury@u2004d01:~$ sudo kubectl logs -n storageos-operator storageos-cluster-operator-57797fcdb-lcr9m
2021-12-02T13:04:04.294Z INFO storageos.setup Initializing {"goversion": "go1.15.2", "os": "linux", "arch": "amd64", "operator-sdk": "v0.18.1"}
2021-12-02T13:04:04.339Z INFO leader Trying to become the leader.
2021-12-02T13:04:04.340Z DEBUG k8sutil Found namespace {"Namespace": "storageos-operator"}
2021-12-02T13:04:04.394Z DEBUG k8sutil Found podname {"Pod.Name": "storageos-cluster-operator-57797fcdb-lcr9m"}
2021-12-02T13:04:04.426Z DEBUG k8sutil Found Pod {"Pod.Namespace": "storageos-operator", "Pod.Name": "storageos-cluster-operator-57797fcdb-lcr9m"}
2021-12-02T13:04:04.432Z INFO leader Found existing lock with my name. I was likely restarted.
2021-12-02T13:04:04.432Z INFO leader Continuing as the leader.
2021-12-02T13:04:04.494Z INFO controller-runtime.metrics metrics server is starting to listen {"addr": "0.0.0.0:8383"}
2021-12-02T13:04:04.551Z DEBUG k8sutil Found namespace {"Namespace": "storageos-operator"}
2021-12-02T13:04:04.563Z INFO storageos.setup Registering Components
2021-12-02T13:04:04.571Z INFO storageos.cluster Adding cluster controller {"k8s": "1.22.3"}
2021-12-02T13:04:04.572Z DEBUG k8sutil Found namespace {"Namespace": "storageos-operator"}
2021-12-02T13:04:04.572Z DEBUG kubemetrics Starting collecting operator types
2021-12-02T13:04:04.572Z DEBUG kubemetrics Generating metric families {"apiVersion": "storageos.com/v1", "kind": "Job"}
2021-12-02T13:04:04.627Z INFO storageos.setup Could not generate and serve custom resource metrics {"error": "discovering resource information failed for Job in storageos.com/v1: apiVersion storageos.com/v1 and kind Job not found available in Kubernetes cluster"}
2021-12-02T13:04:04.681Z DEBUG k8sutil Found namespace {"Namespace": "storageos-operator"}
2021-12-02T13:04:04.682Z DEBUG k8sutil Found podname {"Pod.Name": "storageos-cluster-operator-57797fcdb-lcr9m"}
2021-12-02T13:04:04.690Z DEBUG k8sutil Found Pod {"Pod.Namespace": "storageos-operator", "Pod.Name": "storageos-cluster-operator-57797fcdb-lcr9m"}
2021-12-02T13:04:04.703Z DEBUG metrics Pods owner found {"Kind": "Deployment", "Name": "storageos-cluster-operator", "Namespace": "storageos-operator"}
2021-12-02T13:04:05.126Z INFO metrics Metrics Service object updated {"Service.Name": "storageos-cluster-operator-metrics", "Service.Namespace": "storageos-operator"}
2021-12-02T13:04:05.127Z DEBUG k8sutil Found namespace {"Namespace": "storageos-operator"}
2021-12-02T13:04:05.181Z INFO storageos.setup Could not create ServiceMonitor object {"error": "no ServiceMonitor registered with the API"}
2021-12-02T13:04:05.181Z INFO storageos.setup Install prometheus-operator in your cluster to create ServiceMonitor objects {"error": "no ServiceMonitor registered with the API"}
2021-12-02T13:04:05.181Z INFO storageos.setup Starting the StorageOS Operator
2021-12-02T13:04:05.182Z INFO controller-runtime.manager starting metrics server {"path": "/metrics"}
2021-12-02T13:04:05.182Z INFO controller-runtime.controller Starting EventSource {"controller": "job-controller", "source": "kind source: /, Kind="}
2021-12-02T13:04:05.182Z INFO controller-runtime.controller Starting EventSource {"controller": "nfsserver-controller", "source": "kind source: /, Kind="}
2021-12-02T13:04:05.182Z INFO controller-runtime.controller Starting EventSource {"controller": "storageoscluster-controller", "source": "kind source: /, Kind="}
2021-12-02T13:04:05.182Z INFO controller-runtime.controller Starting EventSource {"controller": "storageosupgrade-controller", "source": "kind source: /, Kind="}
2021-12-02T13:04:06.039Z ERROR controller-runtime.source if kind is a CRD, it should be installed before calling Start {"kind": "Job.storageos.com", "error": "no matches for kind \"Job\" in version \"storageos.com/v1\""}
github.com/go-logr/zapr.(*zapLogger).Error
/go/src/github.com/storageos/cluster-operator/vendor/github.com/go-logr/zapr/zapr.go:128
sigs.k8s.io/controller-runtime/pkg/source.(*Kind).Start
/go/src/github.com/storageos/cluster-operator/vendor/sigs.k8s.io/controller-runtime/pkg/source/source.go:117
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func1
/go/src/github.com/storageos/cluster-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:140
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start
/go/src/github.com/storageos/cluster-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:175
sigs.k8s.io/controller-runtime/pkg/manager.(*controllerManager).startLeaderElectionRunnables.func1
/go/src/github.com/storageos/cluster-operator/vendor/sigs.k8s.io/controller-runtime/pkg/manager/internal.go:514
2021-12-02T13:04:06.040Z DEBUG controller-runtime.manager leader-election runnable finished {"runnable type": "*controller.Controller"}
2021-12-02T13:04:06.040Z ERROR storageos.setup Fatal error {"error": "no matches for kind \"Job\" in version \"storageos.com/v1\""}
github.com/go-logr/zapr.(*zapLogger).Error
/go/src/github.com/storageos/cluster-operator/vendor/github.com/go-logr/zapr/zapr.go:128
main.fatal
/go/src/github.com/storageos/cluster-operator/cmd/manager/main.go:164
main.main
/go/src/github.com/storageos/cluster-operator/cmd/manager/main.go:160
runtime.main
/usr/local/go/src/runtime/proc.go:204