k8s_kubespray - henk52/knowledgesharing GitHub Wiki

k8s Kubespray

Introduction

Purpose

Describe how to configure and use kubespray to automatically deploy and update a kubernetes cluster.

Vocabulary

References

Overview

kubespray uses ansiable to do the automatic deployment and upgrade.

Configuration

Inventory

calico_rr see Optional : BGP Peering with route reflectors
- At large scale you may want to disable full node-to-node mesh in order to optimize your BGP topology and improve calico-node containers' start times.
- calico requires epel to be enabled if you want to use wireguard See Wireguard Encryption in Calico

Installation

Sample installation on four nodes

Deploy the four nodes
ssh node1
sudo -i
ensure the nodes can communicate with each other.
- If required: update the ~/.ssh/authorized_keys in each node.
- ssh -i ~/.ssh/id_rsa node1
- ssh -i ~/.ssh/id_rsa node2
- ssh -i ~/.ssh/id_rsa node3
- ssh -i ~/.ssh/id_rsa node4
- disable the firewall on all nodes:
  - systemctl stop firewalld
  - TODO disable
  - or
    - firewall-cmd --permanent --add-rich-rule='rule family="ipv4" source address="10.26.101.0/24" port protocol="tcp" port="2379-2380" accept'
    - for all ports?
    - TODO what about other apps like postgresql etc?
cd
python -m venv venv
source venv/bin/activate
git clone https://github.com/kubernetes-sigs/kubespray.git
cd kubespray/
git checkout v2.29.1
pip install -r requirements.txt
cp -rfp inventory/sample inventory/mycluster
vi inventory/mycluster/hosts.yaml
- aws ec2 describe-instances --filters Name=instance-state-name,Values=running --query 'Reservations[].Instances[].{InstanceId:InstanceId,PrivateIP:PrivateIpAddress,Name:Tags[?Key==Name]|[0].Value}' --output table
vi inventory/mycluster/group_vars/k8s_cluster/k8s-cluster.yml
- add: kube_version: 1.34.3
  - see the list of supported versions in ./roles/kubespray_defaults/vars/main/checksums.yml
tmux
- ctrl-b + "
- ctrl-b arrow-up
ansible-playbook -i inventory/mycluster/hosts.yaml -u root -b -v --private-key=~/.ssh/id_rsa cluster.yml 2>&1 > /var/log/ansible-cluster-install.log
- this takes about 15 minutes.
ctrl-b arrow-down
tail -f /var/log/ansible-cluster-install.log
when done
- verify installation
  - /usr/local/bin/kubectl get nodes

all:
  hosts:
    node1:
      ansible_host: 10.26.101.128
      ip: 10.26.101.128
      access_ip: 10.26.101.128
    node2:
      ansible_host: 10.26.101.179
      ip: 10.26.101.179
      access_ip: 10.26.101.179
    node3:
      ansible_host: 10.26.101.215
      ip: 10.26.101.215
      access_ip: 10.26.101.215
    node4:
      ansible_host: 10.26.101.67
      ip: 10.26.101.67
      access_ip: 10.26.101.67
  children:
    kube_control_plane:
      hosts:
        node1:
        node2:
    kube_node:
      hosts:
        node1:
        node2:
        node3:
        node4:
    etcd:
      hosts:
        node1:
        node2:
        node3:
    k8s_cluster:
      children:
        kube_control_plane:
        kube_node:
    calico_rr:
      hosts: {}

((venv) ) [root@ip-10-26-101-8 kubespray]# ansible-playbook -i inventory/mycluster/hosts.yml -u root -b -v --private-key=~/.ssh/id_rsa cluster.yml  2>&1 > /var/log/ansible-cluster-install.log
[WARNING]: Could not match supplied host pattern, ignoring: bastion
[WARNING]: raw module does not support the environment keyword
[WARNING]: Platform linux on host node1 is using the discovered Python
interpreter at /usr/bin/python3.12, but future installation of another Python
interpreter could change the meaning of that path. See
https://docs.ansible.com/ansible-
core/2.17/reference_appendices/interpreter_discovery.html for more information.
[WARNING]: Platform linux on host node2 is using the discovered Python
interpreter at /usr/bin/python3.12, but future installation of another Python
interpreter could change the meaning of that path. See
https://docs.ansible.com/ansible-
core/2.17/reference_appendices/interpreter_discovery.html for more information.
[WARNING]: Platform linux on host node3 is using the discovered Python
interpreter at /usr/bin/python3.12, but future installation of another Python
interpreter could change the meaning of that path. See
https://docs.ansible.com/ansible-
core/2.17/reference_appendices/interpreter_discovery.html for more information.
[WARNING]: Platform linux on host node4 is using the discovered Python
interpreter at /usr/bin/python3.12, but future installation of another Python
interpreter could change the meaning of that path. See
https://docs.ansible.com/ansible-
core/2.17/reference_appendices/interpreter_discovery.html for more information.

Trobuleshooting

Unable to look up a name or access an attribute in template string

It looks like this happens when setting the kube_version to 1.34.3, in kubespray v2.29.1

kube version 1.33.7 works fine.

fatal: [node1]: FAILED! => {"msg": "The task includes an option with an undefined variable.. Unable to look up a name or access an attribute in template string ({{ download_cache_dir }}/{{ download.dest | basename }}).\nMake sure your variable name does not contain invalid characte
rs like '-': expected str, bytes or os.PathLike object, not AnsibleUndefined. expected str, bytes or os.PathLike object, not AnsibleUndefined\n\nThe error appears to be in '/root/kubespray/roles/download/tasks/download_file.yml': line 16, column 5, but may\nbe elsewhere in the file
 depending on the exact syntax problem.\n\nThe offending line appears to be:\n\n\n  - name: Download_file | Set pathname of cached file\n    ^ here\n"}