kubernetes deploy tool kustomize helm kubes - ghdrako/doc_snipets GitHub Wiki

• kubectl
• kustomize
• helm
• kubes
• ytt

kustomize

• https://kubernetes.io/docs/tasks/manage-kubernetes-objects/kustomization/
• https://kubectl.docs.kubernetes.io/references/kustomize/
• https://www.innoq.com/en/blog/2022/04/kustomize-introduction
• https://www.innoq.com/en/blog/2022/07/advanced-kustomize-features

Kustomize is a CLI tool, integrated within kubectl tool to manage, customize and apply Kubernetes resources in a template-less way.

With Kustomize, you need to set a base directory with standard Kubernetes resource files (no special characters or placeholders are required) and create a kustomization.yaml file where resources and customizations are declared. A kustomization.yaml file controls the process. Any directory that contains a kustomization.yamlbfile is called a root.

Kustomize has the concept of bases and overlays.

• A base is a directory with a kustomization.yaml file, which contains a set of Kubernetes resources such as deployments, configMaps, services, etc., and associated customization.
• Overlays directories contain environment-specific changes. It may be a name prefix, replica count, a new image tag, or other changes.

kubectl apply --dry-run=client -o yaml -k ./ 
kustomize build
kustomize build . | kubectl apply -f - # apply output generated by kustomize to the cluser

kustomize build overlays/production # see if production values are being applied
kubectl apply -k overlays/production # apply your overlays to the cluster

Kustomize supports the usage of JSON Patch to modify any Kubernetes resource defined as a Kustomize resource. To use it, you need to specify the JSON Patch expression to apply and which resource to apply the patch.

Kustomization:

• kustomize is a Kubernetes feature that uses a file with the name kustomization.yaml to store instructions for the changes a user wants to make to a set of resources
• This is convenient for applying changes to input files that the user does not control himself, and which contents may change because of new versions appearing in Git
• So, it's a way to modify existing configuration files, and thus implement changes in a declarative way
• Use kubectl apply -k ./ in the directory with the kustomization.yaml and the files it refers to to apply changes
• Use kubectl delete -k ./ in the same directory to delete al that was created by the Kustomization

Generating Configuration

• kustomize can also be used to generate configuration
  • use secretGenerator to generate a Secret
  • use configMapGenerator to generate a ConfigMap

Best practice

• Keep your custom resources and their instances in separate packages, otherwise you will encounter race conditions and your creation will get stuck.ex CertManager CRD and CertManager’s resources
• Organize your resources by kind (e.g., horizontal-pod-autoscaler.yaml, service.yaml)
• Try to keep the common values like namespace, common metadata in the base file.
• Follow standard directory structure, using bases/ for base files and patches/ or overlays/ for environment-specific files.
• While developing or before pushing to git, run kubectl kustomize cfg fmt file_name to format the file and set the indentation right.

Example

• https://antonputra.com/kubernetes/kubernetes-kustomize-tutorial/#kustomize-image-name-or-tag
• https://github.com/wings-software/harness-docs/tree/main/kustomize/helloWorld

Basic

Kustomize uses an entry file called kustomization.yaml to manage a collection of resource files. You can manually create this file in the same directory as your resource yaml files or use the following command:

kustomize init

It will generate a basic file kustomization.yaml with the following content:

$cat kustomization.yaml
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization

in directory we have:

$cat pod.yaml
apiVersion: v1
kind: Pod
metadata:
  name: app
spec:
  containers:
  - name: app
    image: nginx:1.14.2
    ports:
    - containerPort: 80

$cat service.yaml
apiVersion: v1
kind: Service
metadata:
  name: app
spec:
  selector:
    app: app
  ports:
  - port: 80
    targetPort: 80

These resource can then be referenced in the kustomization.yaml.

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
- pod.yaml
- service.yaml

Now we can:

kustomize build

The output is a combined rendered yaml document with all the referenced resources. If we want to apply all resources managed by Kustomize directly to a Kubernetes cluster, we can run:

kustomize build | kubectl apply -f -

In comparison to the pure yaml files the output of kustomize brings some advantages:

• The resources are reformatted and combined in a specific order
• This means the output is always reproducible regardless of what order we add the resources in the kustomization.yaml or if we use different formats or indentations in the different yaml files
• The output can then be saved in a new file e.g. to be used in a GitOps Workflow Note The file reference to recource in kustomization.yaml is not limited to local files but can also reference files which are available via http(s), e.g. if they are located in a different repository or are provided by a 3rd party instance.

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
- https://raw.githubusercontent.com/kubernetes/ingress-nginx/controller-v1.1.1/deploy/static/provider/cloud/deploy.yaml

Buildin tranformers

• https://kubectl.docs.kubernetes.io/references/kustomize/builtins/

ReplicaCountTransformer

• Usage via kustomization.yaml
• field name: replicas Replicas modified the number of replicas for a resource.

AnnotationTransformer

• Usage via kustomization.yaml
• field name: commonAnnotations Adds annotations (non-identifying metadata) to all resources. Like labels, these are key value pairs.

commonAnnotations:
  oncallPager: 800-555-1212

ConfigMapGenerator

• Usage via kustomization.yaml
• field name: configMapGenerator

ConfigMaps and Secrets

Kustomize provides some generators to create a ConfigMap or Secret resource for us based on some input key/values.

In the simplest form we can use a generator and just inline the key/values in the kustomization.yaml like this:

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
configMapGenerator:
- name: app-cm
  literals:
  - MY_CONFIG_1=config one
  - MY_CONFIG_2=config two

With this configuration, Kustomize will generate a ConfigMap resource when running the build command and we get this result:

> kustomize build

apiVersion: v1
data:
  MY_CONFIG_1: config one
  MY_CONFIG_2: config two
kind: ConfigMap
metadata:
  name: app-cm-624tfbcc9t

Generated resource has the name as defined in the kustomization.yaml but with an added suffix. This fingerprint is based on the hashed content of the ConfigMap and is per default activated in Kustomize. This can help to avoid accidentally overriding an existing ConfigMap. It would also automatically restart all pods linked to the ConfigMap as the reference changes. If we reference the ConfigMap in other resources, Kustomize takes care that the suffix is added correctly in all these places.

Add config map to pod.yaml:

apiVersion: v1
kind: Pod
metadata:
  name: app
spec:
  containers:
  - name: app
    envFrom:
    - configMapRef:
        name: app-cm
    image: nginx:1.14.2
    ports:
    - containerPort: 80

we see the reference to the ConfigMap gets automatically updated in the Pod definition - with specific suffix

> kustomize build

apiVersion: v1
data:
  MY_CONFIG_1: config one
  MY_CONFIG_2: config two
kind: ConfigMap
metadata:
  name: app-cm-624tfbcc9t     
---
apiVersion: v1
kind: Pod
metadata:
  name: app
spec:
  containers:
  - envFrom:
    - configMapRef:
        name: app-cm-624tfbcc9t
    - secretRef:
        name: app-secret-78585fhggh
    image: nginx:1.14.2
    name: app
    ports:
    - containerPort: 80

SecretGenerator

• Usage via kustomization.yaml
• field name: secretGenerator Each entry in the argument list results in the creation of one Secret resource (it’s a generator of n secrets).

create a Secret by using the secretGenerator

Let’s create a Secret but instead of inline key/values, we let Kustomize read them from a properties file. The properties file secrets.properties look like this:

MY_SECRET=very secret

We then can configure the generator in kustomization.yaml to read the properties

secretGenerator:
- name: app-secret
  envs:
  - secrets.properties

When running the build we get the following output

> kustomize build

apiVersion: v1
data:
  MY_CONFIG_1: config one
  MY_CONFIG_2: config two
kind: ConfigMap
metadata:
  name: app-cm-624tfbcc9t
---
apiVersion: v1
data:
  MY_SECRET: dmVyeSBzZWNyZXQ=
kind: Secret
metadata:
  name: app-secret-78585fhggh
type: Opaque
---
apiVersion: v1
kind: Pod
metadata:
  name: app
spec:
  containers:
  - envFrom:
    - configMapRef:
        name: app-cm-624tfbcc9t
    - secretRef:
        name: app-secret-78585fhggh
    image: nginx:1.14.2
    name: app
    ports:
    - containerPort: 80

Additionally, there are configuration parameters that we can set to manipulate the generation of ConfigMaps and Secrets. A block can be added to the kustomization.yaml like this:

...
generatorOptions:
  labels: # adds labels to the generated resources
    my.label: mylabel
  annotations: # adds annotations to the generated resources
    my.annotation: myannotation
  disableNameSuffixHash: true # disables the suffix generation
  immutable: true # marks the resources as immutable

ImageTagTransformer

• Usage via kustomization.yaml field name: images Images modify the name, tags and/or digest for images without creating patches.

Override image tags and replica count

NamespaceTransformer

• Usage via kustomization.yaml
• field name: namespace Adds namespace to all resources

namespace: my-namespace

PrefixSuffixTransformer

• Usage via kustomization.yaml
• field names: namePrefix, nameSuffix

Modify Prefix, Suffix and Namespace

LabelTransformer

• Usage via kustomization.yaml
• field name: commonLabels Adds labels to all resources and selectors

Add Labels and Annotations

Modify resources for each environment

split our resource definitions into shared definitions (base) used by all environments and patches (overlays) used for specific environments.

The function making these above changes are called transformers in Kustomize, and they work out-of-the-box for resources that are part of the standard Kubernetes API like Deployment and StatefulSet.

Additional CLI commands

A typical case is the image tag that we want to use for the deployment. Let’s say we use the current git commit SHA to tag the docker image of our application. This information is only available when we commit and push the change. So we cannot set the image tag upfront statically in our kustomization.yaml.

For these use cases, Kustomize provides some specific CLI commands that we can use to make some modifications on-the-fly in a pipeline. For the image problem above, we can do the following:

kustomize edit set image app=app:$(git rev-parse --short HEAD)

This dynamically sets the image tag in the kustomization.yaml. Let’s say that the commit hash is 68b4c528 , the new kustomization.yaml would look like this

...
images:
- name: app
  newName: app
  newTag: 68b4c528

The edit command is quite powerful. We can:

• set values for annotations, labels, images, prefix, suffix, namespace and replicas
• dynamically add new base resources, ConfigMaps, Secrets or other resources
• remove resources dynamically which would normally be rendered by Kustomize

Advenced kustomize

• https://github.com/innoq/advanced-kustomize-examples

PatchesStrategicMerge

• Usage via kustomization.yaml
• field name: patchesStrategicMerge

PatchTransformer

• Usage via kustomization.yaml
• field name: patches

Remove a resource from rendering

• https://github.com/kubernetes/community/blob/master/contributors/devel/sig-api-machinery/strategic-merge-patch.md There are multiple directives:
• replace
• merge
• delete
• delete from primitive list

Example

$cat base/app.yaml
apiVersion: v1
kind: Pod
metadata:
  name: myapp
  labels:
    name: myapp
spec:
  containers:
  - name: myapp
    image: app
    ports:
    - containerPort: 8080
---
apiVersion: v1
kind: Service
metadata:
  name: myapp
spec:
  selector:
    app: myapp
  ports:
  - port: 8080
    targetPort: 8080

$cat base/kustomization.yaml
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
- app.yaml

$cat overlay/kustomization.yaml
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
- ../base
patches:
- patch: |-
    $patch: delete
    apiVersion: v1
    kind: Service
    metadata:
      name: myapp

$ kustomize build
apiVersion: v1
kind: Pod
metadata:
  labels:
    name: myapp
  name: myapp
spec:
  containers:
  - image: app
    name: myapp
    ports:
    - containerPort: 8080

Reuse Kustomize configuration

• https://kubectl.docs.kubernetes.io/guides/config_management/components/

components folder contain are commonly use components:

.
├── base
│   ├── deployment.yaml
│   └── kustomization.yaml
├── components
│   └── common-annotations
│       └── kustomization.yaml
├── overlay-dev
│   ├── kustomization.yaml
│   └── service.yaml
└── overlay-prod
    ├── kustomization.yaml
    └── service.yaml

The common-annotations component looks like this:

$cat components/common-annotations/kustomization.yaml
apiVersion: kustomize.config.k8s.io/v1alpha1
kind: Component
commonAnnotations:
  team: my-team

We can reference it in our overlays like this then:

$cat overlay-dev/kustomization.yaml
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
- ../base
- service.yaml
components:
- ../components/common-annotations

When we then extend the component with additional annotations, it will automatically be picked up by all overlays.

Components can contain everything a normal Kustomize configuration can contain, such as:

• image transformers
• patches
• additional resources
• prefix and suffix

Limit labels and annotations to specific resources or fields

commonLabels changes not only the metadata.labels field, but also the selector fields of a service and deployment. In many cases, we want the selector fields untouched anyway and only add labels to the resources metadata.labels.

• https://kubectl.docs.kubernetes.io/references/kustomize/kustomization/labels/

To define just a subset of resources, we can then define an own LabelTransformer (the same works for annotations).

Let’s say we want to add an annotation and a label, but only to the metadata and only the Pod resources, we can define our own transformers like this:

$cat transformers.yaml
apiVersion: builtin
kind: LabelTransformer
metadata:
  name: notImportantHere
labels:
  team: team-a
fieldSpecs:
- kind: Pod
  path: metadata/labels
  create: true
---
apiVersion: builtin
kind: AnnotationsTransformer
metadata:
  name: notImportantHere
annotations:
  team: team-a
fieldSpecs:
- kind: Pod
  path: metadata/annotations
  create: true

The specification is the same as for other transformer configurations. create: true means that metadata.annotations or metadata.labels will be created if they do not exist.

We then add it to our configuration:

$cat kustomization.yaml
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
transformers:
- transformers.yaml
resources:
- app.yaml

When we render the resources, we see that annotation and label is only added to the pod.

json patches

With JSON patches, we can:

• add any field
• replace any field
• copy any field
• move any field
• remove any field

One common need is when we want to modify a list field by, e.g., adding a new entry at the end of the list. This is normally not possible with overlays, as we have to redefine the full list in the overlay again.