Fix Openshift SNO startup problem - mazsola2k/openshift GitHub Wiki
You may encountered that intalled Openshift stable-4.18 either through assisted install method your own cli agent method - openshift worked well until the first restart. Regardless you gracefully restart or accuidentally, the current Openshift SNO not starting up.
Let's review some recovery steps:
sudo bash
By default within the openshift host the oc client won't work until you setup the location for the kubeconfig:
export KUBECONFIG=/etc/kubernetes/static-pod-resources/kube-apiserver-certs/secrets/node-kubeconfigs/lb-int.kubeconfig
Optional chekcs:
oc get csr
oc get csr --no-headers | awk '/Pending/ && /kubelet-serving/ {print }' | xargs -n1 oc adm certificate approve
oc get csr
Steps for debug:
Check the SNO node status:
[root@openshift-worker core]# oc get nodes
NAME STATUS ROLES AGE VERSION
openshift NotReady control-plane,master,worker 46h v1.31.8
[root@openshift core]# oc get pods -n openshift-ingress -owide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
router-default-5df8d99fbb-n54fb **0/1 Pending ** 0 19h <none> <none> <none> <none>
Mandatory steps:
[root@openshift core]# oc adm uncordon openshift
node/openshift uncordoned
After the Openshift startup continues:
oc get nodes
oc get clusteroperators
[root@openshift-worker core]# oc get csr
NAME AGE SIGNERNAME REQUESTOR REQUESTEDDURATION CONDITION
csr-4b5ws 4h38m kubernetes.io/kube-apiserver-client-kubelet system:serviceaccount:openshift-machine-config-operator:node-bootstrapper <none> Pending
csr-7b6b2 13h kubernetes.io/kube-apiserver-client-kubelet system:serviceaccount:openshift-machine-config-operator:node-bootstrapper <none> Pending
csr-7xgz2 112s kubernetes.io/kube-apiserver-client-kubelet system:serviceaccount:openshift-machine-config-operator:node-bootstrapper <none> Pending
csr-b4s8l 13h kubernetes.io/kube-apiserver-client-kubelet system:serviceaccount:openshift-machine-config-operator:node-bootstrapper <none> Pending
csr-ctq5j 14h kubernetes.io/kube-apiserver-client-kubelet system:serviceaccount:openshift-machine-config-operator:node-bootstrapper <none> Pending
csr-dc7dn 14h kubernetes.io/kube-apiserver-client-kubelet system:serviceaccount:openshift-machine-config-operator:node-bootstrapper <none> Pending
csr-flfpn 32m kubernetes.io/kube-apiserver-client-kubelet system:serviceaccount:openshift-machine-config-operator:node-bootstrapper <none> Pending
csr-gzb2c 17m kubernetes.io/kube-apiserver-client-kubelet system:serviceaccount:openshift-machine-config-operator:node-bootstrapper <none> Pending
csr-h6gvd 13h kubernetes.io/kube-apiserver-client-kubelet system:serviceaccount:openshift-machine-config-operator:node-bootstrapper <none> Pending
csr-lsccj 4h23m kubernetes.io/kube-apiserver-client-kubelet system:serviceaccount:openshift-machine-config-operator:node-bootstrapper <none> Pending
csr-nqw9c 4h54m kubernetes.io/kube-apiserver-client-kubelet system:serviceaccount:openshift-machine-config-operator:node-bootstrapper <none> Pending
csr-qv782 4h7m kubernetes.io/kube-apiserver-client-kubelet system:serviceaccount:openshift-machine-config-operator:node-bootstrapper <none> Pending
csr-vc8lt 15h kubernetes.io/kube-apiserver-client-kubelet system:serviceaccount:openshift-machine-config-operator:node-bootstrapper <none> Pending
csr-vhz4s 14h kubernetes.io/kube-apiserver-client-kubelet system:serviceaccount:openshift-machine-config-operator:node-bootstrapper <none> Pending
csr-zwf9t 14h kubernetes.io/kube-apiserver-client-kubelet system:serviceaccount:openshift-machine-config-operator:node-bootstrapper <none> Pending
[root@openshift-worker core]# oc get csr --no-headers | awk '/Pending/ && /kube-apiserver-client-kubelet/ {print $1}' | while read csr; do
> [ -n "$csr" ] && oc adm certificate approve "$csr"
> done