This page provides step-by-step operational procedures for common tasks.

For a failed or unhealthy pod, first inspect its events, current/previous logs, owning Deployment/StatefulSet, ArgoCD reconciliation and related PVCs. Check node conditions and Ceph health when symptoms involve storage or scheduling. Record the observed cause and the smallest recovery action that addresses it.

Review that concrete action and its data impact with the operator before changing the live cluster. Namespace deletion, application reset, PVC removal and cold-start cleanup are explicit recovery procedures; a bad pod state alone does not authorize them. For Nextcloud migration failures, follow the migration recovery boundaries and preserve the current database, writer barrier and private evidence.

1. Stage 1 — Wipe (destructive):

• Set in ansible/config.yml: cold_start_stage_1_wipe_cluster: true and optionally perform_physical_disk_wipe: true.
• Run: ansible-playbook cleanup.yml.

2. Stage 2 — Install Infrastructure:

• Set: cold_start_stage_2_install_infrastructure: true.
• Run: ansible-playbook main.yml.

3. Stage 3 — Deploy Applications:

• Set: cold_start_stage_3_install_applications: true and enable per-app manual_install_* flags.
• Run: ansible-playbook main.yml.

Use only after diagnosis and explicit approval of the selected application and PVC/data handling. Review every enabled cleanup switch before invoking the cleanup playbook.

1. In ansible/config.yml set run_pod_cleanup: true.
2. In pod_cleanup_list, keep only the target app and choose delete_pvc: true|false.
3. Run: ansible-playbook cleanup.yml.
4. Redeploy via Stage 3 if desired.

1. Create a new Intermediate CA signed by the offline Root CA.
2. Update cert-manager ClusterIssuer to reference the new Intermediate.
3. Trigger re-issuance of app certificates.
4. Distribute (or ensure trust of) the Root CA on client devices.

1. Image OS, set static DHCP reservation, ensure SSH.
2. Join as worker with k3s token; verify with kubectl get nodes.
3. Label/taint as needed and confirm workloads schedule as expected.

1. Drain control plane node; upgrade; uncordon.
2. Sequentially drain/upgrade worker nodes; uncordon each.
3. Verify node readiness and workload recovery.

• Prefer accessing UIs via Ingress + TLS at Traefik VIP.
• Check node-exporter and kube-state-metrics.

• Do not commit real credentials. Use Ansible Vault for secrets and GitHub Actions secrets for CI.
• If secrets were previously committed, rotate immediately and remove from repo history if necessary.

Don't hand-edit /config/.storage/lovelace.dashboard_<name> directly — HA owns that file and can overwrite an edit made while it's live. Use deployments/home-assistant/ha_dashboard_edit.py (alongside the HA deployment manifests it operates on), which calls the same lovelace/config / lovelace/config/save WebSocket API the frontend itself uses:

1. Create a Long-Lived Access Token: HA profile (bottom-left) → Security → Long-Lived Access Tokens.
2. Find the dashboard's real url_path (not necessarily its title): kubectl exec -n home-assistant home-assistant-0 -c home-assistant -- cat /config/.storage/lovelace_dashboards.
3. Edit NEW_CARDS in the script for whatever card(s) you're adding.
4. kubectl cp deployments/home-assistant/ha_dashboard_edit.py home-assistant/home-assistant-0:/tmp/ha_dashboard_edit.py -c home-assistant
5. kubectl exec -n home-assistant home-assistant-0 -c home-assistant -- env HA_TOKEN="<token>" HA_DASHBOARD_URL_PATH="<url_path>" python3 /tmp/ha_dashboard_edit.py
6. Revoke the token from the same Security screen once done, if it was only needed for this edit.