issue: kube‐apiserver not running - dcasota/photonos-scripts GitHub Wiki

The issue entry https://github.com/vmware/photon/issues/1558 attracted my attention, because quite often users get 'connection refused' messages in kubectl get nodes with the latest Photon OS packages.

Photon OS repository contains the packages kubernetes, kubernetes-kubeadm, kubernetes-dns, kubernetes-metrics-server, kubernetes-pause and kubernetes-dashboard. Unfortunately there are issues with the utility options. Mixes from options is actually a challenge.

  • option1 - cpu architecture: x86_64, arm64
  • option2 - device: brigded, nat, mirrored
  • option3 - cgroupv2 support (no/yes)
  • option4 - configure systemd on boot
  • option5 - enable TLS on etcd
  • option6 - install docker as: root, non-root user, rootless
  • option7 - install/update package release version using: tdnf, vendor website
  • option8 - install tools kubernetes: kubeadm, minikube, kubectl
  • option9 - run cluster as: regular user, root user
  • option10 - configured overlay
  • option11 - Photon OS updates
  • option12 - network event brokering and cluster reconfiguration
  • ..

The user guide chapters 'Running Kubernetes on Photon OS' and 'Kubeadm Cluster on Photon OS' with 'configuring a Master Node' do not help in any constellation.

The following recipe is an attempt to configure a working setup and to discuss culprits.

Setup Preparation - Configure ntp

Login as root.

NTPSERVER1="ntp11.metas.ch"
NTPSERVER2="ntp12.metas.ch"
NTPSERVER3="ntp13.metas.ch"

tdnf install -y ntp iptables

cat <<EOFSERVERS | tee -a /etc/ntp.conf
server $NTPSERVER1
server $NTPSERVER2
server $NTPSERVER3
EOFSERVERS

iptables -A INPUT -i eth0 -p udp --dport 123 -j ACCEPT
iptables-save >/etc/systemd/scripts/ip4save
systemctl enable ntpd
systemctl start ntpd

ln -sf /usr/share/zoneinfo/Europe/Zurich /etc/localtime

systemctl stop ntpd
ntpdate -s $NTPSERVER1
systemctl start ntpd
hwclock --systohc --localtime

Setup Stage - Configure docker

Login as root and install docker.

# https://vmware.github.io/photon/docs-v5/administration-guide/containers/docker-rootless-support/
tdnf install -y docker
systemctl stop docker
systemctl enable docker
systemctl enable containerd

To start docker on boot with systemd there is a setting necessary.

# https://docs.docker.com/engine/install/linux-postinstall/#configure-docker-to-start-on-boot-with-systemd
# https://docs.docker.com/reference/cli/dockerd/
mkdir -p /etc/docker
# for root
cat <<EOFOPTS | tee /etc/docker/daemon.json
{
  "exec-opts": ["native.cgroupdriver=systemd"]
}
EOFOPTS

systemctl start containerd
systemctl start docker

Setup Stage - Configure rootless docker

Run the docker daemon as non-root user (rootless mode) enhances security.

Let's configure this.

# https://vmware.github.io/photon/docs-v5/administration-guide/containers/docker-rootless-support/
# https://docs.docker.com/engine/security/rootless/

tdnf install -y shadow fuse slirp4netns libslirp
tdnf install -y docker-rootless

# https://docs.docker.com/engine/install/linux-postinstall/
# Manage docker as a non-root user
ROOTLESS_USER="rootless" # change here
# subuid start range is prepared for systemd-homed, see https://rootlesscontaine.rs/getting-started/common/subuid/#specific-subuid-range-for-systemd-homed-users
uid="524288"
# -f -1               : disable password expiration
# -m                  : home folder
# -g docker           : primary group
useradd $ROOTLESS_USER -f -1 -m -g docker
echo "$ROOTLESS_USER:$uid:65536" >> /etc/subuid
echo "$ROOTLESS_USER:$uid:65536" >> /etc/subgid
echo "kernel.unprivileged_userns_clone = 1" >> /etc/sysctl.d/50-rootless.conf
chmod 644 /etc/subuid /etc/subgid /etc/sysctl.d/50-rootless.conf
# reload
sysctl --system
modprobe ip_tables

# change password of rootless-user
passwd $ROOTLESS_USER

# Configure login initialization for rootless-user
cat <<EOFROOTLESS | tee /tmp/rootless.sh
echo 'export PATH=/usr/local/bin:$PATH' >> ~/.bash_profile
echo 'export DOCKER_HOST=unix:///run/user/$(id -u)/docker.sock' >> ~/.bash_profile
EOFROOTLESS
chmod 644 /tmp/rootless.sh
chmod a+x /tmp/rootless.sh
ssh ${ROOTLESS_USER:+${ROOTLESS_USER}@}localhost '/tmp/rootless.sh && exit'
rm -f /tmp/rootless.sh

# Run precheck as rootless-user
ssh ${ROOTLESS_USER:+${ROOTLESS_USER}@}localhost 'dockerd-rootless-setuptool.sh check --force && exit'

# Run installation as rootless-user
ssh ${ROOTLESS_USER:+${ROOTLESS_USER}@}localhost 'dockerd-rootless-setuptool.sh install --force && exit'
# If it fails, check docker.service: ssh ${ROOTLESS_USER:+${ROOTLESS_USER}@}localhost 'journalctl --user -xefu docker.service && exit'

Add a demo user <>root and <>rootless-user. Exit root.

DEMO_USER="dcaso"
useradd $DEMO_USER -f -1 -m -g users -G sudo,wheel
passwd $DEMO_USER

exit

Setup Stage - Configure Kubernetes

Login as demo user.

sudo tdnf install -y apparmor-parser kubernetes-kubeadm kubernetes kubernetes-dns kubernetes-metrics-server kubernetes-pause

# Iptables ports master
sudo iptables -A INPUT -p tcp -m tcp --dport 2379:2380 -j ACCEPT # etcd server client API
sudo iptables -A INPUT -p tcp -m tcp --dport 6443 -j ACCEPT # Kubernetes API Server
sudo iptables -A INPUT -p tcp -m tcp --dport 10250:10252 -j ACCEPT # Kubelet API, kube-scheduler and kube-controller-manager
sudo iptables -A INPUT -p tcp -m tcp --dport 10255 -j ACCEPT # Read-Only Kubelet API

sudo iptables -A INPUT -p icmp --icmp-type echo-request -j ACCEPT
sudo iptables -A INPUT -p tcp -m tcp --dport 2375 -j ACCEPT
sudo iptables -A INPUT -p tcp -m tcp --dport 179 -j ACCEPT
sudo iptables -A INPUT -p tcp -m tcp --dport 4789 -j ACCEPT

sudo iptables-save | sudo tee -a /etc/systemd/scripts/ip4save

# for user
mkdir -p $HOME/.docker
cat <<EOFOPTS | tee $HOME/.docker/daemon.json
{
  "exec-opts": ["native.cgroupdriver=systemd"]
}
EOFOPTS

sudo systemctl daemon-reload

# https://kubernetes.io/docs/setup/production-environment/container-runtimes/#prerequisite-ipv4-forwarding-optional
echo 'net.ipv4.ip_forward = 1' | sudo tee -a /etc/sysctl.d/10-ip_forward.conf
sudo sysctl -w net.ipv4.ip_forward=1

cat <<EOFKCONF | sudo tee -a /etc/sysctl.d/kubernetes.conf
net.ipv4.ip_forward = 1
net.bridge.bridge-nf-call-ip6tables = 1
net.bridge.bridge-nf-call-iptables = 1
net/bridge/bridge-nf-call-arptables = 1
EOFKCONF

cat <<EOFCCONF | sudo tee -a /etc/modules-load.d/containerd.conf
br_netfilter
overlay
EOFCCONF

sudo modprobe br_netfilter
sudo modprobe overlay
sudo sysctl --system

# for non-root user accounts
mkdir -p $HOME/.kube
sudo cp --force /etc/kubernetes/kubeconfig $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config

# Initialize
sudo kubeadm init

# Check nodes
kubectl get nodes

Troubleshooting

Analyze kubectl issues

kubectl get nodes -v10

Analyze ports

sudo tdnf install -y netcat
nc -z -v <ip> <port>

Fix etcd.service issues

sudo systemctl stop etcd
sudo rm -rf /var/lib/etcd
sudo tdnf reinstall etcd

Configure Minikube

cd $HOME
sudo usermod -aG docker $USER && newgrp docker
sudo systemctl start docker
curl -Lo minikube https://storage.googleapis.com/minikube/releases/latest/minikube-linux-aarch64
chmod +x minikube
./minikube delete --all=true --purge=true
./minikube start --embed-certs
⚠️ **GitHub.com Fallback** ⚠️