oneks_monitor_troubleshooting - OpenNebula/cluster-api-provider-opennebula GitHub Wiki

OneKS Monitor Troubleshooting

See OneKS Monitor Overview for an introduction to the component.

Quick checks

Start by checking all monitor resources and recent log messages:

kubectl -n kube-system get helmchart oneks-monitor
kubectl -n kube-system get deployment oneks-monitor
kubectl -n kube-system get pods \
  -l app.kubernetes.io/name=oneks-monitor
kubectl -n kube-system logs deployment/oneks-monitor --tail=200

The HelmChart does not exist

Confirm that:

  1. The monitor section is configured in oneks-server.conf.
  2. The OneKS configuration passed validation when the service started.
  3. The cluster was created with the monitor feature enabled.
  4. The generated workload-cluster manifests contain the oneks-monitor HelmChart.

The chart coordinates should be:

Repository: https://opennebula.github.io/cluster-api-provider-opennebula/charts/
Chart:      oneks-monitor

The HelmChart exists but the Deployment does not

Inspect the HelmChart and the Helm controller jobs:

kubectl -n kube-system describe helmchart oneks-monitor
kubectl -n kube-system get jobs
kubectl -n kube-system get pods

Common causes include:

  • The configured chart version does not exist.
  • The workload cluster cannot reach the CAPONE Helm repository.
  • The cluster cannot download the image from ghcr.io.
  • The RKE2 Helm controller is unavailable.

The monitor Pod does not start

Inspect the Deployment and Pod events:

kubectl -n kube-system describe deployment oneks-monitor
kubectl -n kube-system describe pod \
  -l app.kubernetes.io/name=oneks-monitor

Then inspect the logs:

kubectl -n kube-system logs deployment/oneks-monitor --previous

Configuration errors detected during startup include:

  • Missing or invalid OneKS endpoint.
  • Missing or invalid OpenNebula XML-RPC endpoint.
  • Missing authentication Secret or Secret key.
  • Authentication data that is not in username:password-or-token form.
  • An invalid per-cluster encryption key.
  • Invalid polling intervals.

The Pod is running but not Ready

Readiness requires every monitor component to complete its initial Kubernetes cache synchronization or resource listing.

Check the logs for Kubernetes API or RBAC errors:

kubectl -n kube-system logs deployment/oneks-monitor

You can also inspect the health endpoints locally:

kubectl -n kube-system port-forward deployment/oneks-monitor 8081:8081

In another terminal:

curl --fail http://127.0.0.1:8081/healthz
curl --fail http://127.0.0.1:8081/readyz

healthz confirms that the process is running. readyz confirms that the initial Kubernetes caches and listings are ready. Readiness does not guarantee current connectivity to OpenNebula or OneKS.

OneKS does not receive updates

Check the monitor logs for callback or OpenNebula lookup errors:

kubectl -n kube-system logs deployment/oneks-monitor \
  | grep -E 'endpoint|authentication|resolve|send|XML-RPC'

Verify that the workload cluster can reach:

  • The configured OneKS callback endpoint.
  • The configured OpenNebula XML-RPC endpoint.

Also confirm that:

  • The OneKS cluster identifier is correct.
  • The monitor and OneKS use the same per-cluster encryption key.
  • The mounted credential is valid.
  • The OpenNebula VM metadata identifies the expected OneKS cluster and Node Group.

The monitor retries transient delivery and OpenNebula lookup failures.

Application states are not updated

List the OneKS-managed HelmChart resources:

kubectl -n kube-system get helmcharts \
  -l oneks.opennebula.io/managed=true

Only resources in kube-system with the following label are monitored:

oneks.opennebula.io/managed: "true"

They must also contain the OneKS annotations generated by the application service, including the release name and lifecycle state.

Resource observations are missing

Inspect the configuration:

kubectl -n kube-system get configmap oneks-resource-monitor -o yaml

Check that:

  1. monitor.yaml contains a YAML array.
  2. resource is the plural API resource name, not the Kubernetes kind.
  3. apiVersion, namespace, name, and field path are correct.
  4. The selected value is scalar, not an object or array.
  5. The monitor ServiceAccount can read the resource.

Review the logs for parsing, lookup, or permission errors:

kubectl -n kube-system logs deployment/oneks-monitor \
  | grep observation

An invalid update does not replace the last valid observation configuration. If one configured resource cannot be read, the monitor avoids publishing a partial snapshot.

Related documentation