Kubernetes

Introduction

References

Hec18 - Kubernetes for developers by Joseph Heck
Say20 - Mastering kubernetes third edition by Gigi Sayfan, 2020
kubectl cheatsheet
k8s tutorials
k8s docker cluster
k8s the hard way
logging
kubernetes.io docs
k8s on slack
cncf
meetup cncf
Intro to helm
Install minkube
- minikube on github
Micro k8s

Kubernetes: native tools

kubectl config get-contexts
- switch kubectl between clusters
kubectl config use-context minikube

Azure kubernetes - AKS

AKS
- Also look at the 'Concepts' entry

Network

Security:

Training:

Nan21 - The Ultimate Kubernetes Administrator Course | CKA

TODO look at these pages:

Vocabulary

Dashboard
- Dashboard Documents
- Project page
- Dashboard Wiki
- Dashboard access control
ConfigMap - External configuraiont of your application(, 1.2).
DaemonSet - Automatically calcluate how many replicas are needed based on the number of existing nodes(, 1.2)
Deployment -
- stateless apps(, 1.2)
Deployment controller - primarily responsible for rolling out software updates and managing the process of that rollout when you update your deployment resource with new versions of your software(Hec18,p19).
ephemeral - lasting a very short time
Federation - sync accross clusters
- cluster federation in k8s
- set-up cluster federation
headless service - a service that returns the IP of every attached pod, rather than just a random one great description
helm - installation and management of kubernetes apploications.
- Helm chart
- Stable charts
Ingress -
kubeadm - used for setting up single master clusters.
kubectl - command interface
kubernetes manifest - yaml configureation file(s). Declarative
- vs kubectl is imperative
Kompose - tool to convert Docker Compose files into Kubernetes objects.
kops - Kubernetes Operations.
- Allow you to create, destroy, maintain kubernetes clusters from the command line.
Maturity - ??? Kubernetes Operators Explained
- 1 - basic install
- 2 - upgrades
- 3 - lifecycle
- 4 - insights
- 5 - auto-pilot
minikube - VM with a master in it, for testing out stuff.
Namespaces - Pods are collected into namespaces, which are used to group Pods together for a variety of purposes(Hec18,p16).
Node - a machine(physical or VM) that has been added to the Kubernetes cluster(Hec18,p17).
Operator - automate handling of statefulsets Kubernetes Operator simply explained in 10 mins
pod - made up of one or more containers and information assuciated with those container(Hec18, p15) Like affinity, how k8s should react if the container fails
- Smalletst unit in kubernetes(,1.2)
PV - Persistent Volume
PVC - Persistent Volume Claim
ReplicaSet - wraps Pods, defining how many need to run in parallel(Hec18,p19). represent horizontal scaling
- A ReplicaSet is associated with a Pod and indicates how many instances of that Pod should be running within the cluster(Hec18,p19).
SC - Storage Class
Secret - Used to store secret data. Stored in base64. Meant to be encrypted by 3rd party app.
service - permanent IP address and load balancer.
StatfulSet - meant for e.g. databases(, 1.2)
terraform -
Volumes - local or remote storage, e.g. outside k8s cluster
- k8s doesn't manage data persistance(, 1.2).

Commands

kubectl config get-contexts
kubectl config use-context minikube
kubectl create -f helloworld-all.yml
kubectl expose deployment helloworld --type=NodePort
kubectl get all
kubectl delete ...
- kubectl delete deploy/hw
- kubectl delete service/hw
- kubectl delete deployment helloworld-all-deployment
- kubectl delete service helloworld-all-service
- kubectl delete pods --selector dev-lead=karthik
kubectl get deployments
- kubectl get deployment hw -o yaml
kubectl get nodes
kubectl get pods
- kubectl get pod helloworld-deployment-with-bad-liveness-probe-575d9d65c5-l546s
- kubectl get pods --selector env=production --show-labels
- kubectl get pods --selector dev-lead=karthik,env=staging --show-labels
- kubectl get pods --selector dev-lead!=karthik,env=staging --show-labels
- kubectl get pods --selector 'release-version in (1.0,2.0)' --show-labels
  - show for either 1.0 or 2.0
- kubectl get pods --selector 'release-version notin (1.0,12.0)' --show-labels
- kubectl get pods --show-labels
- kubectl get pods -L run,pod-template-hash(Hec18,p71)
kubectl get rs
kubectl get services
kubectl label ...
- kubectl label pod helloworld app=helloworldapp --overwrite
  - This seems to overwrite if 'app' exists not all labels.
- kubectl label pod helloworld app-
  - Delete the 'app' label.
kubectl scale --replicas=3 deploy/helloworld-all-deployment
- kubectl scale --help
- kubectl scale --record=true
  - kubectl rollout history deployment nginx-deployment
    - shows the recorde history.
kubectl set image deployment navbar-deployment helloworld=karthequian/helloworld:blue
kubectl rollout ...
- kubectl rollout history deployment navbar-deployment
- kubectl rollout undo deployment navbar-deployment
  - --to-revision
kubectl run helloworld --image=karthequian/helloworld --port=80

basic Troubleshooting

kubectl describe deployment
kubectl describe pod
kubectl logs helloworld-deployment-with-bad-liveness-probe-575d9d65c5-l546s
kubectl exec -it helloworld-deployment-with-bad-liveness-probe-575d9d65c5-l546s /bin/bash
kubectl exec -it helloworld-deployment-with-bad-liveness-probe-575d9d65c5-l546s -c helloworld /bin/bash
- If you have multiple containers in a pod.
minikube addons list
minikube dashboard
minikube addons enable heapster
kc get pods --all-namespaces
kubectl apply -f red.yaml
kubectl create configmap logger --from-literal=log_level=debug
- Configmap name: logger
- key: log_level
- value: debug
kubectl create namespace NAME_SPACE
kubectl create URL
kubctl delete pods -n cart --all
kubctl delete pods -n cart --all --grace-period=0 --force
kubectl get
- kubectl get configmaps
- kubectl get configmap/logger -o yaml
- kubectl get componentstatuses
- kubectl create secret generic apikey --from-literal=api_key=123456789
- kubectl get secrets
- kubectl get secret apikey -o yaml
- kubectl get jobs
- kubectl get cronjobs
- kubectl get daemonsets
- kubectl get nodes --show-labels
- kubectl get statefulsets
- kubectl get namespaces
- kubectl get deployments -o wide
- kubectl get pods -o wide --all-namespaces
- kubectl get pods -o json --all-namespaces
- kubectl get pods --sort-by=.metadata.name --all-namespaces
- kubectl get pods -o=jsonpath="{..image}" -l env=staging -n cart
  - jsonpath
kubectl proxy - opens up a proxy to the Kubernets REST API(Hec18,p43)

kubectl get sorting by field

kubectl get pods --sort-by='{metadata.uid}'

kubectl get formating output

kubectl get pod -o jsonpath='{.items[0].metadata}' | jq
kubectl get pod -o jsonpath='{.items[*].metadata.name}'
kubectl get pod -o jsonpath='{range .items[*]}{.metadata.name}{"\n"}{end}'
kubectl get pod -o jsonpath='{range .items[*]}{.metadata.name}{"\t"}{.status.podIP}{"\n"}{end}'
kubectl get pod -o jsonpath='{range .items[*]}{.metadata.name}{"\t"}{.status.podIP}{"\t"}{.status.startTime}{"\n"}{end}'
- (Nan20, 6.4)
- write these jsonpath ops as scripts
kubectl get pod -o custom-columns=POD_NAME:.metadata.name
kubectl get pod -o custom-columns=POD_NAME:.metadata.name,POD_IP:.status.podIP,CREATE_AT:.status.startTime

Installing kubectl

See: Install using native package management

sudo apt update
sudo apt install -y ca-certificates curl
sudo curl -fsSLo /etc/apt/keyrings/kubernetes-archive-keyring.gpg https://packages.cloud.google.com/apt/doc/apt-key.gpg
echo "deb [signed-by=/etc/apt/keyrings/kubernetes-archive-keyring.gpg] https://apt.kubernetes.io/ kubernetes-xenial main" | sudo tee /etc/apt/sources.list.d/kubernetes.list
sudo apt update
sudo apt install -y kubectl

switching between multiple clusters using kubectl context

(Nan20, 17.2)

the cubeconfig.yaml contains one or more clusters and one or more users
context is a combination of a cluster and a user
`current-context:' references the context that is currently active(Nan20, 17.2)
- This is the context used when executing kubectl commands(Nan20, 17.2).
kubectl config --help
switch the current context: kubectl config use-context <VALID_CONTEXT>
set the namespace to use for a context kubectl config set-context --curent --namespace=kube-system
- TODO how do I get my current/active namespace

apiVersion: v1
kind: Config
preferences: {}
clusters:
- cluster:
    certificate-authority-data: xxxxxx
    server: https://172.31.44.88:6443
  # The name we use for the cluster
  name: development
- cluster:
    certificate-authority-data: xxxxxx
    server: https://251.15.20.12:6443
  name: staging  
contexts:
- context:
    cluster: development
    user: dev-admin
    # you can also specify a namespace
  name: dev-admin@development
- context:
    cluster: development
    namespace: myapp
    user: my-script
  name: my-script@development
- context:
    cluster: staging
    user: staging-admin
  name: staging-admin@staging
current-context: dev-admin@development
users:
  # This is the reference name we use in this file for the user being defined
- name: dev-admin
  user:
    client-certificate-data: xxxx
    client-key-data: xxxx
- name: staging-admin
  user:
    client-certificate-data: xxxx
    client-key-data: xxxx 
- name: my-script
  user:
    token: xxxxxxx

mkdir ~/.k8s_certs
chmod 750 ~/.k8s_certs
cd ~/.k8s_certs
mkdir minkube_isrock
cd minkube_isrock
scp in the three files (you can find them referenced in ~/.kube/config)
- ~/.minikube/ca.crt
- ~/.minikube/profiles/minikube/client.crt
- ~/.minikube/profiles/minikube/client.key
kubectl config set-cluster isrock_minkube --certificate-authority=/home/USER/.k8s_certs/minkube_isrock/ca.crt --server=https://192.168.1.144:6443
kubectl config set-credentials isrock-admin --client-certificate=/home/USER/.k8s_certs/client.crt --client-key=/home/USER/.k8s_certs/client.key
kubectl config set-context isrock --user=isrock-admin --cluster=isrock_minkube --namespace=default
kubectl config use-context isrock
kubectl config current-context

couldn't get current server API group list: Get "https://192.168.1.144:6443/api?timeout=32s": tls: failed to verify certificate: x509: certificate is valid for 192.168.49.2, 10.96.0.1, 127.0.0.1, 10.0.0.1, not 192.168.1.144

Deploying from private registry

Overview

TODO make a SwArcDoc like overview

Keynote: Kubernetes by Kelsey Hightower, Staff Developer Advocate, Google
API
Cloud Controller Manager(CCM)
kubelet
Scheduler
Kube Controller manager
kube proxy
kube dns
etcd
NodePort

Things to add

Helm
logging
policies
metrics
storage
security

k8s concepts Breakdown

Ingress - a Kubernetes configuration object that lets you expose a service to the outside world
- The 'ingress' is used for defining the rules that the 'Ingress controller' will apply/implement.
- An API object that manages external access to the services in a cluster, typically HTTP.
- It seems that your need an 'Ingress Controller' to get this functionality(Say20,p148).
Ingress controller
- minikube: minikube addons enable ingress
Service - an abstraction which defines a logical set of Pods running somewhere in your cluster, that all provide the same functionality.
- This allows for a single point of contact that will loadbalance and handle pods restarting with new IP addresses.
- The Service API lets you expose an application running in Pods to be reachable from outside your cluster.
- Create the service before the backend workloads and workloads that need access.
  - or else the environment variables will not be populated.
workloads or or several components.
- Deployments - ?
  - Ensures pods are re-created if a node goes down.
  - If you have a "naked" pod (without deployment or ReplicaSet) then if a node goes down, the pod will not be recreated somewhere else.
  - TODO what is the purpose of having deployments of replicasets?
- ReplicaSet - used to manage stateless applications and ensure a specified number of replicas are running
- StatefulSet - used to manage stateful applications and ensure stable, unique network identities and predictable pod creation and deletion.
  - StatefulSets are valuable for applications that require one or more of the following.
    - Stable, unique network identifiers.
    - Stable, persistent storage.
    - Ordered, graceful deployment and scaling.
    - Ordered, automated rolling updates.
- DaemonSet
- Job
- CronJob
Node
- Worker
  - container runtime[mandatory]
    - containerd (much more lightweight than docker)(Nan21, 1.3)
    - cri-o
    - docker
  - kubelet[mandatory] - inteacts with both the container and the node(Nan21, 1.3).
    - starts the containers
    - allocate resources to the container
  - kube proxy[mandatory] -
- Master - aka control plane nodes
  - There are usually multiple master nodes(Nan21, 1.3).
  - API server[mandatory] - cluster gateway
    - act as getkeeper for authentications(Nan21, 1.3)
  - Scheduler[mandatory] -
    - where to put the pod
  - controller manager[mandatory] - detect cluster state changes
    - recover crashes, by talking to the scheduler
  - etcs[mandatory] - key value store(Nan21, 1.3)
    - cluster brain(Nan21, 1.3)
    - cluster changes get stored in the key value store(Nan21, 1.3)
    - No application data.

Nana's quick tip

alias k=kubectl(Nan20, 3.11)
use kubctl commands to generate a yaml file
- kubectl create service clusterip test-new-cidr --tcp=80:80 --dry-run=client -o yaml > my-svc.yaml
kubectl create service --help

managing k8s resources; kubectl vs yaml

kubectl - imperative(Nan21, 1.4)
- practical when testing
- quick one-off tasks
- just getting started
yaml - declarative(Nan21, 1.4)
- infrastructure as code
- history f configurations
- collaboration and review process possible
- more transparent(TODO how?)

three parts of a configuration file

Metadata - metadata:
Specification - `spec:
status(which is what k8s will attain)(Nan21, 1.5)
- kubernetes gets this status information from the etcd.(Nan21, 1.5)

Namespaces

You can't access most resources from another Namespace(Nan20, 2.12)
- ConigMap
- Secrets
You can access Service in another Namespace (append Namespace to name: mysql.namespace)(Nan20, 2.12)
not bound to a namespace(Nan20, 2.12)
- kubectl api-resources --namespaced=false
- vol
- node
- ...

use cases when to use Namespaces

Group your applications in namespaces(Nan20, 2.12)
- Structure your components
One namespace per team(Nan20, 2.12)
- Avoid conflicts between teams
split staging and development(Nan20, 2.12)
- share services between different environments
Blue/Green deployment(Nan20, 2.12)
Access and reources limits on namespaces(Nan20, 2.12)
- Access and reouserce limits

Labels

Kubernetes project uses short, definitive key/value pairs as set of tags on resources, called labels(Hec18,p66).
- There is a matching mechanism to query and find related labels, called Selectors(Hec18,p66).
Labels are not intended to identify a single or unique resource(Hec18,p66).
Labels can be used to describe relevant information about a set of Kubernetes resources, be that a Pod, ReplicaSet, Deployment, and so on(Hec18,p66).
Prefixes are optional the prefix and label are seperated with an '/'(Hec18,p67).
- a prefix needs to be 253 characters or less(Hec18,p67)
- Prefixes, are expected to be a DNS domain(Hec18,p67).
- Internal components and plugins to Kubernetes are expected to use prefixes to group and segregate their labels(Hec18,p67)
- kubernetes.io is reserved for Kubernetes internal labels(Hec18,p67)
- If a prefix is not defined, then it is considered entirely under user control(Hec18,p67)
The key beyond the prefix has a maximum length of 63 characters(Hec18,p67).
- Keys can also only be specified with alphanumeric characters, as well as -, _(Hec18,p67)
Labels are intended to represent semantic information about a resource, and having multiple labels is not only acceptable, but expected(Hec18,p67).
Most common are dimensions of interest(Hec18,p67)
- Environment
- Version
- Application name
- Tier of service
Kubernetes does nothing to prevent you from confusing or reusing simplistic labels(Hec18,p68)
You should consider making and maintaining a living document with labels you use and their meaning and intentions(Hec18,p67).
- Hec18 prefer to do this in a README.md in the deploy directory where he keeps the Kubernetes declarations, and find that whatever conventions you set are critical to understand, especially when you are working as part of a team(Hec18,p67).

Selectors

Selectors are used in Kubernetes to connect resources together based on the labels they have (or don’t have)(Hec18,p68). A selector is meant to provide a means to retrieve a set of resources in Kubernetes(Hec18,p68).

Most of the kubectl commands support a -l option that allows you to provide a selector to filter what it finds(Hec18,p68).

A Selector can be(Hec18,p68)
- equality-based to represent specific values. Equality selectors use(Hec18,p68)
  - =
  - !=.
- set-based to allow filtering and selection based on multiple values Set selectors use(Hec18,p68)
  - in
  - notin
  - exists

Deployment

See: Kubernetes YAML File Explained - Deployment and Service | Kubernetes Tutorial 19

apiVersion: apps/v1
kind: Deployment
metadata:
  name: rest-deployment
  labels:
    app: rest
spec:
  replicas: 3
  selector:
    matchLabels:
      app: rest
  template:
    metadata:
      labels:
        app: rest
    spec:
      containers:
      - name: rest
        image: rest_service:0.1.1
        ports:
        - containerPort: 8000
        livenessProbe:
          httpGet:
            path: /liveness
            port: 8000
          initialDelaySeconds: 10
          timeoutSeconds: 1
        readinessProbe:
            httpGet:
              path: /readiness
              port: 8000

selector - specify the labels used for POD selection(Nan20, 3.3)

Replicaset

Nan20, 13.3

Deployment strategies

Recreate strategy - delete the old, create the new. Application downtime (Nan20, 13.3).
Roling update strategy - stop one old pod, start one new pod(Nan20, 13.3)
- Or will it start the new first and then stop one old one, when the new one is working.
each roll-out creates a history
- kubectl rollout history deployment nginx-deployment
kubectl rollout history deployment/{depl-name}
kubectl rollout undo deployment/{depl-name}
kubectl rollout status deployment/{depl-name}

Service

Being able to group together a set of Pods that all do the same thing, so that we can scale them and access them, is what the Kubernetes resource Service is all about(Hec18,p72).

A Service is the Kubernetes resource used to provide an abstraction through to your Pod (or Pods) that is agnostic of the specific instances that are running.

Providing a layer between what one container (or set of containers) provides(Hec18,p72), such as a frontend web application, and another layer, such as a database, allows Kubernetes to(Hec18,p72)
- scale them independently,
- update them,
- handle scaling issues,
- and more.
A service also can contain a policy by which data should be transferred(Hec18,p72),
- so you might consider it a software load balancer within Kubernetes.
A Service is also the key abstraction used to expose Pods to each other(Hec18,p72),
- or your container outside the Kubernetes cluster(Hec18,p72)
A service is the heart of how Kubernetes manages the coordination between sets of Pods, as well as traffic in and out of them(Hec18,p72).

An advanced use of Service also allows you to define a service for a resource entirely outside the cluster. This can allow you to have a consistent means of using services, regardless of whether the endpoint you need to run is from within Kubernetes or external to the cluster.(Hec18,p72)

Kubernetes includes an expose command that can create a service based on a resource that is already operating within the cluster(Hec18,p73).

Service types Publishing Services
- ClusterIP - internal service
  - Most services will define a ClusterIP, and Kubernetes will handle all the dynamics of linking up the resources as(Hec18,p73)
  - when the clusterIP is set to "None" then it is a Headless Services
Service type
Exploring Kubernetes Service Types

Simple service declaration(Hec18,p73)

kind: Service
apiVersion: v1
metadata:
  name: service
spec:
selector:
  run: flask
ports:
- protocol: TCP
  port: 80
  targetPort: 5000

targetPort can be a string referring to the name of a port and not just a port number(Hec18,p74)
- TargetPort references the ContainerPort of the deployment/pod(Nan20, 3.3)
selector - list of key/value pairs that reference the POD labels(Nan20, 3.3)
For each service, there are environment variables defined that provide the IP address, port, and protocol with a couple of name variations(Hec18,p77).
- environment variables are set based on the state of Kubernetes at the time that the Pods are created and they are not updated during the life of the Pod(Hec18,p84).
- Note that this IP address is not the IP address of any underlying Pods, but an IP address within the Kubernetes cluster that the service is managing as a single endpoint for accessing the selected Pods(Hec18,p77).
A DNS entry is created and coordinates with every service defined, so that you can request the DNS entry for <service> or <service>.<namespace>, and the internal DNS services will provide you with a correct internal IP address(Hec18,p78).
- acking on a namespace should only be done when you are explicitly trying to refer to a service in another namespace(Hec18,p78).
in general it’s best to always define and apply your service declarations first(Hec18,p77).
- Ordering is critical with services!(Hec18,p77)
  - If Pods exist prior to the Service being defined, then the environment variables for that service will not exist within those Pods(Hec18,p77).
  - Restarting the Pods, or scaling them down to 0 and back up (forcing the containers to be killed and recreated) will resolve it(Hec18,p77).
Service is not a process(Nan20, 3.3)
- kube-proxy forwards the request(Nan20, 3.3)
- kube-proxy is responsible for maintaining the list of Service IPs and coresponding Pod IPs(Nan20, 3.3)
- Whe the request is sent to the service IP then kube-proxy will intercept the request, kube-proxy will check its lists for which pods are registered for this service IP address and the pick one of the pods
kubectl get svc nginx-service -o yaml

Endpoints

TODO what is an 'Endpoint'??? Endpoint defines the network location of a service that runs outside the k8s cluster(Hec18,p74). e.g. a service that you are still running as a VM in Azure. Using the Endpoint, it seems like by creating n enpoint you can make the VM is accessible via the endpoint.

If you are migrating services into Kubernetes and have some of those services external to the cluster, this provides one way to represent the remote system as a service internally, and if you move it into Kubernetes later, you wouldn't have to change how internal Pods connect or utilize that resource(Hec18,p74).

A service does not require a selector, and a service without a selector is how Kubernetes represents a service that's outside of the cluster for the other resources within(Hec18,p74).

kind: Service
apiVersion: v1
metadata:
  name: some-remote-service
spec:
  ports:
  - protocol: TCP
    port: 1976
    targetPort: 1976

kind: Endpoints
apiVersion: v1
metadata:
  name: some-remote-service
subsets:
  - addresses:
    - ip: 1.2.3.4
  ports:
    - port: 1976

ExternalName
- There is a variant to the Endpoint definition that simply provides a DNS reference, called an ExternalName service(Hec18,p75).
- ExternalName simply defines an external DNS entry that can be used as a service definition(Hec18,p75).
- ExternalName only provides a DNS response and does not manage any port forwarding or redirection(Hec18,p75).

kind: Service
apiVersion: v1
metadata:
 name: another-remote-service
 namespace: default
spec:
 type: ExternalName
 externalName: my.rest.api.example.com

Service type – NodePort

ok for testing
Use loadbalancer instead
Not user friendly
Insecure and messy
exposes the service definition through a high-numbered port on all of the Kubernetes cluster nodes(Hec18,p79)
- 30000 - 32767(Nan20, 4.2)
NodePort also creates ClusterIP interal Service(Nan20, 4.2)
When you are using a Kubernetes cluster on premises, or in our case in a virtual machine on your development machine with Minikube, NodePort is a common service type used to expose your services(Hec18,p79).
NodePort relies on the underlying hosts upon which you run Kubernetes to be accessible on your local network, and exposes the service definition through a high-numbered port on all of the Kubernetes cluster nodes(Hec18,p79).
These services are exactly like the default ClusterIP services, with the exception that they have a type of NodePort(Hec18,p79).

apiVersion: v1
kind: Service
metadata:
  name: nginx-service
  labels:
    app: nginx
    svc: test-nginx
spec:
  type: NodePort
  selector:
    app: nginx
  ports:
    - protocol: TCP
      port: 8080
      targetPort: 80
      nodePort: 30001

Default type is 'Cluster' so only internally accessible.

apiVersion: v1
kind: Service
metadata:
  creationTimestamp: 2017-10-14T18:19:07Z
  labels:
    run: flask
  name: flask
  namespace: default
  resourceVersion: "19788"
  selfLink: /api/v1/namespaces/default/services/flask
  uid: 2afdd3aa-b10c-11e7-b586-080027768e7d
spec:
  type: NodePort
  clusterIP: 10.0.0.39
  externalTrafficPolicy: Cluster
  ports:
  - nodePort: 31501
    port: 5000
    protocol: TCP
    targetPort: 5000
  selector:
    run: flask
  sessionAffinity: None
status:
  loadBalancer: {}

Service type - LoadBalancer

Runs outside the k8s cluster(Nan20, 4.3)
Coordinates with the cloud provider's infrastructure to set up an external LoadBalancer that will forward traffic into the service(Hec18,p79). The LoadBalancer service type is not supported in all Kubernetes clusters(Hec18,p79).
- AWS supports
  - HTTP(S)
  - TCP/TLS/UDP
  - IP
How you define these services is specific to your cloud provider, and slightly different between AWS, Azure, and Google(Hec18,p79).
On a self-managed k8s cluster, you have to create our own LoadBalancer(Nan20, 4.3)
With Ingress active in your cluster, you only need a single LoadBalancer(Nan20, 4.4)
Usually the Cloud provider have the LoadBalancer that you need(Nan20, 4.4)

Headless service - no IP

Headless service is when you set the ClusterIP to 'None', if there is a reason that you want to definitively control what specific pods you connect and communicate with(Hec18,p76).

kind: Service
apiVersion: v1
metadata:
  name: flask-service
spec:
  ClusterIP: None
  selector:
    app: flask

Other stuff

Database

DB can't be replicated via deployment, due to has a state(, 1.2)
DB are often hosted outside of the k8s cluster, due to the challanges of persistency(, 1.2)
or use statefulsets

Statefulsets

Kubernetes Tutorial: Why Do You Need StatefulSets in Kubernetes?

When you define a number of replicas, those will be started up in sequence, not parallel.
- pod-0 will be started first
- pod-1 will be started when pod-0 is fully operational(readiness probe reports ok)
stateful sets are closed down in reverse order, in sequence
- sort pod-2 will be stopped first
- when pod-2 is completely removed, pod-1 will be stopped and removed.
This sequential way of starting and stopping is probably to support DB and DB replication
- pod-0 will be the master; and will be able to both read and write to the database.
- pod-1 will be the first db replica; it will only have read access and it will be replicated from pod-0
- pod-2 will be a db replica, it will only have read acess and it will be replicated from pod-1 when starting up, then sync from master
  - pod-2 is replicating the db from pod-1 to reduce the network load on pod-0
  - once the db has been completely replicated from pod-1, then pod-2 is set to sync from pod-0
- The replica pods knows the address of the master(pod-0)
  - we can't depende on the IP address since that is randomly assigned, but in stateless, the pod names are static: <pod_name>-<instance_id>

Operators

multiple containers in a deployment

Sidecar Container

The container providing helper functionality is called a sidecar container(Nan20, 7.2)
Usually the sidecar operates asynchronously.
e.g. db sync jobs
sidecar containers are continers that run the whole time your main container is running(Nan20, 7.2).
Starts at the same time as the main container(Nan20, 7.2)
Containers in the same pod are able to talk to each other using localhost(Nan20, 7.2)
You can have multiple sidecar containers in a pod.

init containers

Run once in the beginning and exits(Nan20, 7.2)
The main container startes after the init container has finished(Nan20, 7.2)
Init container are used to initialize something inside your pod(Nan20, 7.2)

Containers waiting for init

 k get pod
NAME                                        READY   STATUS     RESTARTS      AGE
debug-pod                                   1/1     Running    1 (21h ago)   21h
ingress-nginx-controller-5c6fd54c59-r56mq   1/1     Running    1 (44h ago)   14d
nginx-deployment-75d4684fd5-5ktm4           0/2     Init:0/1   0             6s
nginx-deployment-75d4684fd5-q8bxg           0/2     Init:0/1   0             6s
nginx-deployment-75d4684fd5-s4z6l           0/2     Init:0/1   0             6s
test-nginx-svc                              1/1     Running    1 (44h ago)   15d

get it unstuck with: kubectl create service clusterip mydb-service --tcp=80:80 (Though it did not work for me)

(Nan20, 7.3)

apiVersion: apps/v1
kind: Deployment
metadata:
  name: nginx-deployment
  labels:
    app: nginx
spec:
  replicas: 3
  selector:
    matchLabels:
      app: nginx
  template:
    metadata:
      labels:
        app: nginx
    spec:
      containers:
      - name: nginx
        image: nginx
        ports:
        - containerPort: 80
      - name: log-sidecar
        image: busybox
        command: ['sh', '-c', "while true; do echo sync app logs; sleep 20; done"]
      initContainers:
      - name: mydb-available
        image: busybox
        command: ['sh', '-c', "until nslookup mydb-service; do echo waiting for database; sleep 4; done"]

Monitoring

Logging

Log to stdout

logs from multicontainer pods
- if the webapp pod have multiple container, on called 'background'(Hec18, p60)
- kubectl logs webapp background
- or kubectl logs webapp -c background
logs from previous containers, if still available: add -p option(Hec18, p61)
Timestamps by adding '--timestamps' to the logs command(Hec18, p61)
- The timestamps are from the docker host, not the container(Hec18, p61).

Health

Nodes health
health of kubernetes
Application health and metrics
cAdvisor
heapster - seems to indirectly collect from cAdvisor
Prometheus
Grafana

liveness probe

For use if a container inside the pod dies(Nan20,12.2) even if the pod seems healthy.
Indicates if the pod is still alive and providing service.
See: conf liveness and rediness probes
There are thre ways to to this(Nan20,12.2)
- exec
  - Expects the return code 0, if things are ok
- httpGet
  - probably expects 200 or something for ok.
- tcpSocket
  - If it succeeds in establishing a connection, then the container is concidered healthy(Nan20,12.2)

e.g.

              livenessProbe:
                exec:
                  command: 
                  - pgrep
                  - sockperf
                initialDelaySeconds: 5

readiness probe

The readinessProbe is meant to indicate whether the container is ready to service requests(Hec18,p16).
- (Nan20,12.2)
readinessProbe supports the same three interfaces as the livenessProbe(Nan20,12.2)

Security

Normal users:
Service accounts:

Def:

Username
UID
Group
Extra fields
Authentication: Does a user have access to the system?
- Client certs
- Static token files
- OpenID connect
- Webhook mode
Authorization: Can the user perform an action in the system?
- ABAC: Attribute-based access control
- RBAC: Role-based access control (recomended to be turned on)
- Webhook (works well with third party authorization tool (e.g. AD))

kubernetes sections

ingress

Ingress

Websocket

websocket

K8s networking

network policies

network policies
networkpolicy
Mastering kubernetes, 2nd edition.

networking accross name spaces

High availability

High availability

Kubernetes the hard way

Installing private k8s clusters

Microk8s

Installing microk8s

See: micro k8s

sudo snap install microk8s --classic
microk8s status --wait-ready
microk8s enable dashboard dns registry istio
microk8s kubectl get all --all-namespaces
microk8s dashboard-proxy

microk8s start
microk8s stop

If RBAC is not enabled access the dashboard using the default token retrieved with:

token=$(microk8s kubectl -n kube-system get secret | grep default-token | cut -d " " -f1)
microk8s kubectl -n kube-system describe secret $token

In an RBAC enabled setup (microk8s enable RBAC) you need to create a user with restricted
permissions as shown in:
https://github.com/kubernetes/dashboard/blob/master/docs/user/access-control/creating-sample-user.md

KinD

See: kind quick start

Installing KinD

go install sigs.k8s.io/[email protected]
kind create cluster
kind get kubeconfig > ~/.kube/kind.config
export KUBECONFIG=~/.kube/kind.config
(kind delete cluster)

kind get kubeconfig > ~/.kube/kind.config
export KUBECONFIG=~/.kube/kind.config
kc cluster-info

Create KinD multi-node

vi .kind/kind-multi-node-config.yaml
kind create cluster --config ~/.kind/kind-multi-node-config.yaml

~/.kind/kind-multi-node-config.yaml

kind: Cluster
apiVersion: kind.x-k8s.io/v1alpha4
nodes:
- role: control-plane
- role: worker
- role: worker

Create KinD HA cluster

cat ~/.kind/kind-ha-multi-node-config.yaml

kind: Cluster
apiVersion: kind.x-k8s.io/v1alpha4
nodes:
- role: control-plane
- role: control-plane
- role: worker
- role: worker

Pods in k8s

3_1

Each pod get an IP address, not the container(s)
Usually 1 app per pod
can have multiple containers
static pods get the name of the node appende to the end(Nan20, 2.13)
- e.g. etcd-master
- kubectl get pod -n kube-system

Pod lifecycle

Pod status field

Pending - k8s has accepted the pod, but is waiting for something else
Running - all is running
Succeeded - pod is finished all containers sttoppped
Failed - all pods stopped but an errror occured
Unknown - don't know but will wait 5 minutes before trying again
roll-back -

Pod information

kubectl get pod -o yaml
- (Nan20, 7.4)
There are two ways to expose pod field to a running container:
- Environment variables
- Volume files

Pod information via ENV vars

(Nan20, 7.4)
Expose Pod Information to Containers Through Environment Variables
It looks to me like the 'fieldPath: metadata.name' is a reference to the json structure?

      - name: log-sidecar
        image: busybox
        command: ['sh', '-c']
        args:
        - while true; do
            echo sync app logs;
            printenv POD_NAME POD_SERVICE_ACCOUNT POD_IP;
            sleep 20;
          done
        env:
        - name: MY_FIXED_ENV
          value: my-value
        - name: POD_NAME
          valueFrom:
            fieldRef:
              fieldPath: metadata.name
        - name: POD_SERVICE_ACCOUNT
          valueFrom:
            fieldRef:
              fieldPath: spec.serviceAccountName
        - name: POD_IP
          valueFrom:
            fieldRef:
              fieldPath: status.podIP

Types of pod controllers

job - for pods that are expeced to terminate.
ReplicationController, ReplicaSet, deployment - pods that are not expected to terminate.
DaemonSet - pods that need to run once per machine.

k8s events sent to the pod

postStart - sent immediatly after a container is started.
preStop - sent immediatly before the container is terminated.

TODO describe what they can be used for

apiVersion: v1
kind: pod
...
spec:
  containers:
  - name: lifcycle-demo-container
    image: nginx
    lifecucle:
      postStart:
        exec:
          command: ["/bin/sh","-c","echo Hello > /usr/share/message"]
      preStop:
        exec:
          command: ["usr/sbin/nginx","-s","quit"]

Pod configuration

You can also have the pod configuration as part of the deployment.yaml

pod livenessProbe - pod has started

There are three types of liveness probes:

httpGet: The probe is considered successful if the HTTP status is between 200 and 399(Say20, p168)
TcpSocket: Just checks that a port is open
Exec: Runs a command that returns 0 for success

apiVersion: apps/v1
kind: Deployment
metadata:
  name: rest-deployment
  labels:
    app: rest
spec:
  replicas: 3
  selector:
    matchLabels:
      app: rest
  template:
    metadata:
      labels:
        app: rest
    spec:
      containers:
      - name: rest
        image: rest_service:0.1.1
        ports:
        - containerPort: 8000
        livenessProbe:
          httpGet:
            path: /liveness
            port: 8000
          initialDelaySeconds: 10
          timeoutSeconds: 1
        readinessProbe:
            httpGet:
              path: /readiness
              port: 8000

pod readynessProbe - pod is ready to handle requests

you can also use readiness probes to temporarily remove pods that are overbooked until they drain some internal queue(Say20, p169).

Provide configuration to Pods/containers

Inject Data Into Applications Kubernetes Patterns: The ConfigMap Pattern How To Create Config-Map Using K8s Python Client?

(Nan20, 9.1)

Pods can sonsume ConfigMaps or Secrets in two different ways
- As individual values(Nan20, 9.1)
  - Using Environment Variables(Nan20, 9.1)
- as configuration files
  - using volumes
The configmap and secret must exist before the deployment is applied(Nan20, 9.2).
Updates to configMaps and secrets do not affect running PODs, you need to restart all the pods(Nan20, 9.3)
- kubectl rollout restart deployment/my-db
  - This will restart all the pods, and they will have the latest information.
  - kubectl rollout status deployment/my-db

Save existing configmap to a yaml file

kubectl get configmap my-configmap -n my-namespace -o yaml > my-configmap.yaml
- kubectl get configmap fluent-bit -n default -o yaml > my-fluent-bit-configmap.yaml

ConfigMap as Environment variables

(Nan20, 9.2)

kubectl apply -f my-configmap.yaml

apiVersion: v1
kind: ConfigMap
metadata:
  name: myapp-config
data:
  db_host: mysql-service

ConfigMap as files

(Nan20, 9.3)

apiVersion: v1
kind: ConfigMap
metadata:
  name: mysql-config-file
data:
  # configure it as a file
  mysql.conf: |
    [mysqld]
    port=3306
    socket=/tmp/mysql.sock
    key_buffer_size=16M
    max_allowed_packet=128M

Secret as Environment variables

kubectl apply -f my-secret.yaml
'Opaque' Secrets have to be base64 encoded(Nan20, 9.2)'
- echo -n "myuser" | base64
  - bXl1c2Vy

apiVersion: v1
kind: Secret
metadata:
  name: myapp-secret
type: Opaque
data:
  username: bXl1c2Vy
  password: bXlwd2Q=

Secret as file