03 Getting Started - SoftwareAG/edge-k8s-operator-docs GitHub Wiki

CAUTION

Eary access release of the Cumulocity IoT Edge Kubernetes Operator is not ready for production deployments. Software AG doesn't support upgrades of the Edge deployments done using this version of the Operator.

The Edge Kubernetes Operator deploys and manages Cumulocity IoT Edge, version 1014.0.x.

The Operator is tested on the following lightweight Kubernetes distributions:

• K3s version 1.21.x
• K3d version 5.2.x

Ideally, the Operator should work on any Cloud Native Computing Foundation (CNCF) certified Kubernetes distribution.

K3s is a certified lightweight Kubernetes distribution for IoT * Edge computing. K3s is packaged as a single <50MB binary that reduces the dependencies and steps needed to install, run and auto-update a production Kubernetes cluster. Refer to Installing K3s for the installation instructions.

K3d is a lightweight wrapper to run K3s in docker. K3d makes it very easy to create single-node K3s clusters in docker, e.g. for local development on Kubernetes. Refer to Installing K3d for the installation instructions.

INFO

If you want to manage the Kubernetes cluster from a remote machine, you will need kubectl installed on your client machine. Refer to Installing kubectl for the installation instructions. Ensure you verify and update the kubectl configuration after installation.

A Helm chart is available for installing the Edge Kubernetes Operator. To begin, create a new single node Kubernetes cluster with the Kubernetes distribution of your choice, and configure kubectl to use that cluster. See the System Requirements page for the supported Kubernetes distributions and versions.

INFO

To use Helm charts, you need to install Helm v3. Refer to Installing Heml for the installation instructions.

For installing the Operator, you need a license key and credentials to access the Cumulocity IoT Edge repository. Please contact Nick Ponomar [email protected], Edge Product Manager, to get the necessary details.

Run this command after replacing <EDGE_REPO_USERNAME> and <EDGE_REPO_PASSWORD> with the Edge repository credentials you received to add the Operator Helm charts repository to your Helm and fetch the charts.

#
# Add the Operator Helm charts repository to your Helm.
#

# Replace <USERNAME> and <PASSWORD> with the credentials you received
EDGE_REPO_USERNAME=<USERNAME>
EDGE_REPO_PASSWORD=<PASSWORD>

# This adds edge chart repository to your helm
helm repo add edge_repo https://registry.c8y.io/chartrepo/edge --username ${EDGE_REPO_USERNAME} --password ${EDGE_REPO_PASSWORD}

# This fetches all the Helm charts available in the repo.
helm repo update

Run this command after replacing <EDGE_REPO_USERNAME> and <EDGE_REPO_PASSWORD> with the credentials you received to deploy the Operator.

INFO

This script depends on ./jq for determining the latest version of the Operator. Refer to Installing jq.

#
# Deploy the Cumulocity IoT Edge Kubernetes Operator.
#

# Replace <USERNAME> and <PASSWORD> with the credentials you received
EDGE_REPO_USERNAME=<USERNAME>
EDGE_REPO_PASSWORD=<PASSWORD>

# Resolves the latest version of the Operator
EDGE_OPERATOR_VERSION=$(helm search repo edge_repo -l -o json | jq -r 'map(select(.name | contains("edge-operator")).version) | max')

# Deploys the Operator
# If necessary, change the Operator name from edge-k8s-operator to a name relevant for you
# Pass --namespace and --create-namespace options to deploy in a specific namespace
helm install edge-k8s-operator edge_repo/edge-operator --version=${EDGE_OPERATOR_VERSION} --username=${EDGE_REPO_USERNAME} --password=${EDGE_REPO_PASSWORD}

By default, the Operator is deployed in edge-k8s-operator-system namespace. However, you can override by providing the --namespace and --create-namespace (to create namespace if not present) options to the command. You can also change the Operator name in the command from edge-k8s-operator to a name relevant for you.

INFO

We recommend not using the Edge Custom Resource (CR) name as Kubernetes Operator's namespace, as it is used for deploying one of the Edge components.

Use kubectl get pods --all-namespaces to check and ensure the status of the edge-k8s-operator-controller-manager pod is "Running" before continuing with the subsequent steps.

To use the Edge Kubernetes Operator to deploy Cumulocity IoT Edge, create a Kubernetes manifest file with an Edge Custom Resource (CR) which describes the Edge deployment. Then use kubectl to apply that configuration file to your Kubernetes cluster.

Before deploying the Cumulocity IoT Edge, you must Install Operator on your Kubernetes cluster using Helm.

For hosting microservices, the Cumulocity IoT Core Platform needs a Docker registry to store and deliver the microservice images to Kubernetes cluster. Hence, the Kubernetes cluster in which you plan to deploy the Edge needs to be configured to connect to this registry (even for the externally hosted Docker registry scenario).

Upon startup, K3s checks to see if a registries.yaml file exists at /etc/rancher/k3s/ and instructs containerd to use any registries defined in the file. You need to create a registries.yaml file with the registry details, and it needs to be created as root on the node that uses the registry.

For connecting to the registry installed by the Operator, you can use registries.yaml. Replace cumulocity-iot-edge in the file with the Edge CR name you plan to deploy.

INFO

In order for the registry changes to take effect, you need to restart K3s on each node.

For more details, refer to Private Registry Configuration

You can add registries by specifying them in a registries.yaml and referencing it at creation time: k3d cluster create mycluster --registry-config "my-registries.yaml".

For more details refer to Using Image Registries

Kubernetes doesn't use its CoreDNS service to resolve hostnames, rather, it uses DNS servers listed in the /etc/resolv.conf of the Operating System. As the Cumulocity IoT Core Platform needs DNS to resolve the microservices registry (Docker registry) host, you need to update /etc/resolv.conf file to add the cluster IP address of the CoreDNS service at the top of the nameserver list.

Execute this command to find the cluster IP of the CoreDNS or kube-dns (K3s cluster) service.

kubectl get services -A   # Use the cluster IP of the CoreDNS or kube-dns service

Add nameserver <cluster IP address of CoreDNS or kube-dns service> to /etc/resolv.conf file.

If you are using Ubuntu 18.04 or later, the resolv.conf file is a symbolic link managed by the Resolve daemon. In this case, you need to disable ResolveD before updating the file.

# Stop ResolveD service
sudo systemctl disable systemd-resolved.service
sudo systemctl stop systemd-resolved

# Remove the symbolic link created by the ResolveD
rm /etc/resolv.conf

# Create a new resolv.conf
touch /etc/resolv.conf

# Use the cluster IP of the CoreDNS or kube-dns service
echo "nameserver <cluster IP address of CoreDNS or kube-dns service>" >> /etc/resolv.conf 

INFO

Refer to Advanced settings configuration in WSL for configuring DNS in Windows Subsystem for Linux (WSL2).

You can use Sample Cumulocity IoT Edge Manifest file with the Edge Custom Resource (CR) definition, Secrets and Ingress necessary to deploy Cumulocity IoT Edge.

Updates needed to the sample manifest file,

• sag-registry-credentials secret to supply the Software AG Registry (registry.c8y.io) credentials to the Cumulocity IoT Edge Operator.
  • Replace the <EDGE_REPO_USERNAME> and <EDGE_REPO_PASSWORD> with the repository credentials you received.
• license-secret secret to supply the Edge license key to the Cumulocity IoT Edge Operator.
  • Replace the <EDGE-LICENSE-KEY> with the license text you received.
• To change the Edge Custom Resource name, replace the occurrences of cumulocity-iot-edge with the name you want to use.

Refer to the Edge Custom Resource Definition for structure and configuration options available in the Edge Custom Resource (CR).

CAUTION

Kubernetes Secrets are, by default, stored unencrypted in the API server's underlying data store (etcd). Anyone with API access can retrieve or modify a Secret, and so can anyone with access to etcd. Additionally, anyone who is authorized to create a Pod in a namespace can use that access to read any Secret in that namespace; this includes indirect access such as the ability to create a Deployment.

Follow the approach recommended by the Kubernetes distribution you are using to secure the secrets.

The CR defined in the sample manifest file deploys Edge version 1014.0.0 named "cumulocity-iot-edge", with the below details

• myown.iot.com domain with self-signed tls certificates
• Cumulocity IoT Core and related pods in cumulocity-iot-edge namespace
• MongoDB in the cumulocity-iot-edge-mongodb namespace
• Docker Registry in the cumulocity-iot-edge-microservices-registry namespace
• Logging components in the cumulocity-iot-edge-logging namespace
• Apama, Smart Rules, Device Simulator and OPCUA Management Server microservices in the cumulocity-iot-edge-microservices namespace
• Administration, Cockpit, Device Management and Apama Streaming Analytics applications

Use the manifest file you updated to deploy Edge.

For example, you can deploy Sample Cumulocity IoT Edge Manifest file with,

kubectl apply -f cumulocity-iot-edge-manifest.yaml

You can edit the manifest file anytime and kubectl apply to manage the Edge deployment.

Use kubectl get pods --all-namespaces to check the pods to confirm the status. It may take several minutes for the pods to provision resources, initialize, and start running. Before you continue, please wait for the pods in the cumulocity-iot-edge* namespaces to switch to the Running state.

Sample output:

NAMESPACE                                    NAME                                                              READY   STATUS      RESTARTS   AGE
kube-system                                  local-path-provisioner-7b7dc8d6f5-86x4p                           1/1     Running     0          16h
kube-system                                  helm-install-traefik-crd-d962m                                    0/1     Completed   0          16h
kube-system                                  helm-install-traefik-jrxgx                                        0/1     Completed   1          16h
kube-system                                  svclb-traefik-00cea91a-chwf9                                      2/2     Running     0          16h
kube-system                                  traefik-7cd4fcff68-6k2zg                                          1/1     Running     0          16h
kube-system                                  coredns-b96499967-w8zkk                                           1/1     Running     0          16h
kube-system                                  metrics-server-668d979685-l9bmx                                   1/1     Running     0          16h
edge-k8s-operator-system                     edge-k8s-operator-controller-manager-749fc495fd-md57k             2/2     Running     0          35m
cumulocity-iot-edge-logging                  logging-operator-6b5fcd96c8-kx7z4                                 1/1     Running     0          25m
cumulocity-iot-edge-logging                  eventtailer-event-tailer-0                                        1/1     Running     0          25m
cumulocity-iot-edge-logging                  logging-fluentd-configcheck-4bdeec00                              0/1     Completed   0          25m
cumulocity-iot-edge-logging                  logging-fluentd-0                                                 2/2     Running     0          25m
cumulocity-iot-edge-logging                  logging-fluentbit-6wj4g                                           1/1     Running     0          25m
cumulocity-iot-edge-mongodb                  edge-mongodb-sharded-configsvr-0                                  1/1     Running     0          25m
cumulocity-iot-edge-mongodb                  edge-mongodb-sharded-mongos-5c958dd955-skzrr                      1/1     Running     0          25m
kube-system                                  svclb-cumulocity-iot-edge-service-mqtt-42efe561-tjr2h             2/2     Running     0          24m
cumulocity-iot-edge-mongodb                  edge-mongodb-sharded-shard0-data-0                                1/1     Running     0          25m
cumulocity-iot-edge-microservices-registry   internal-registry-7f6d784d44-8rt5l                                1/1     Running     0          24m
cumulocity-iot-edge                          c8ycore-sts-0                                                     3/3     Running     0          24m
cumulocity-iot-edge-microservices            smartrule-scope-management-deployment-66c9c6589f-8f8fv            1/1     Running     0          19m
cumulocity-iot-edge-microservices            device-simulator-scope-management-deployment-6797b7fd7d-sdnkc     1/1     Running     0          15m
cumulocity-iot-edge-microservices            apama-ctrl-scope-edge-deployment-7dbd8dfbd9-r7nqx                 1/1     Running     0          11m
cumulocity-iot-edge-microservices            ssl-management-server-scope-management-deployment-67c95c8cskphj   1/1     Running     0          11m

Kubernetes Ingress is an API object that manages external access to the services in a cluster, typically HTTP. You must have an Ingress controller to satisfy an Ingress. Only creating an Ingress resource has no effect. K3s and K3d installations by default deploy Traefik ingress controller.

To connect to Edge, you need to find the IP address on which Ingress is reachable. Use kubectl get service traefik -n kube-system to find the external IP address of the traefik service.

Below is the sample output of the kubectl get service command.

NAME      TYPE           CLUSTER-IP           EXTERNAL-IP             PORT(S)                      AGE
traefik   LoadBalancer   X.X.X.X **REDACTED   X.X.X.X **REDACTED      80:31640/TCP,443:30351/TCP   16h

You can connect to the Edge on this EXTERNAL-IP address.

The Cumulocity IoT Edge is accessible using the domain name configured as part of the deployment in the Edge CR. In the Sample Cumulocity IoT Edge Manifest, we used myown.iot.com.

There are two ways to configure the accessibility with the domain names

1. Add an entry of the domain name and IP mapping in the DNS servers.
2. Add the host name alias in /etc/hosts or C:\Windows\System32\drivers\etc\hosts to access Edge through the domain name. This needs to be performed on each client host from which the Edge is accessed.

INFO

The first option is always preferable so that the Edge is accessible over LAN.

Enter the URL https://myown.iot.com in the browser.

The Edge login screen appears. Log in with your credentials provided during the installation in the admin-credentials-secret of the Sample Cumulocity IoT Edge Manifest.

INFO

Please refer to Device integration using MQTT for connecting your devices over MQTT.

The Operator deploys and configures a Fluent Bit daemonset on the node to collect container and application logs from the node file system. Fluent Bit queries the Kubernetes API, enriches the logs with metadata about the pods (in the Edge Namespaces), and transfers both logs and metadata to Fluentd. Fluentd receives, filters, and persists the logs in the node's file system.

You can find the logs, including the ones from the Edge Operator, at /var/log/cumulocity-iot-edge/<EDGE-CR-NAME>. Logs for the sample deployment can be found under /var/log/cumulocity-iot-edge/cumulocity-iot-edge folder.