NREC instances and domains for Microk8s, Rancher, and Hono API - uib-ub/uib-ub-monorepo GitHub Wiki
For hono api, we have NREC project uib-ub-api in BGO region, and 3 instances (Rocky Linux 9) have been created for Microk8s Kubernetes cluster:
hono_api_prod_n01: 158.37.65.7
hono_api_prod_n02: 158.37.65.60
hono_api_prod_n03: 158.37.65.111
There are also 6 volumes created. Although we might not need to use docker, three of them are mounted to the Docker directory /var/lib/docker:
hono_api_vol_prod_n01 - 30GiB - rbd - /dev/sdb on hono_api_prod_n01
hono_api_vol_prod_n02 - 30GiB - rbd - /dev/sdb on hono_api_prod_n02
hono_api_vol_prod_n03 - 30GiB - rbd - /dev/sdb on hono_api_prod_n03
Another three volumes are mounted to the Microk8s directory /var/snap/microk8s:
microk8s_vol_prod_n01 - 30GiB - rbd - /dev/sdc on hono_api_prod_n01
microk8s_vol_prod_n02 - 30GiB - rbd - /dev/sdc on hono_api_prod_n02
microk8s_vol_prod_n03 - 30GiB - rbd - /dev/sdc on hono_api_prod_n03
In Security Group http, we open Microk8s API server port 16443 for the remote access of the Microk8s kubernetes cluster:
Ingress - IPv4 - TCP - 16443 - 0.0.0.0/0
Also, if needed (currently we don't), we could open ports from 30000 to 32767, used by NordPort of the Kubernetes cluster for testing purposes:
Ingress - IPv4 - TCP - 30000 - 32767 - 129.177.0.0/16
For testing purposes, we have created domains including api-dev.testdu.uib.no, api-test.testdu.uib.no, and api-prod.testdu.uib.no:
> openstack recordset create testdu.uib.no. api-dev --type A --record 158.37.65.7 --record 158.37.65.60 --record 158.37.65.111
+-------------+--------------------------------------+
| Field | Value |
+-------------+--------------------------------------+
| action | CREATE |
| created_at | 2024-05-28T11:39:16.000000 |
| description | None |
| id | 8500cb44-bbe5-4785-87f9-a6a8632afa7d |
| name | api-dev.testdu.uib.no. |
| project_id | 6735f97c792c4bb3b342b920b1f13f02 |
| records | 158.37.65.111 |
| | 158.37.65.7 |
| | 158.37.65.60 |
| status | PENDING |
| ttl | None |
| type | A |
| updated_at | None |
| version | 1 |
| zone_id | f2c1d504-d3cc-4d3c-a602-1940688f4b81 |
| zone_name | testdu.uib.no. |
+-------------+--------------------------------------+
> openstack recordset create testdu.uib.no. api-test --type A --record 158.37.65.7 --record 158.37.65.60 --record 158.37.65.111
+-------------+--------------------------------------+
| Field | Value |
+-------------+--------------------------------------+
| action | CREATE |
| created_at | 2024-05-28T11:39:34.000000 |
| description | None |
| id | ee3caf1a-d8fd-4ff5-bf55-6b696c5e1aab |
| name | api-test.testdu.uib.no. |
| project_id | 6735f97c792c4bb3b342b920b1f13f02 |
| records | 158.37.65.111 |
| | 158.37.65.7 |
| | 158.37.65.60 |
| status | PENDING |
| ttl | None |
| type | A |
| updated_at | None |
| version | 1 |
| zone_id | f2c1d504-d3cc-4d3c-a602-1940688f4b81 |
| zone_name | testdu.uib.no. |
+-------------+--------------------------------------+
> openstack recordset create testdu.uib.no. api-prod --type A --record 158.37.65.7 --record 158.37.65.60 --record 158.37.65.111
+-------------+--------------------------------------+
| Field | Value |
+-------------+--------------------------------------+
| action | CREATE |
| created_at | 2024-05-28T11:39:48.000000 |
| description | None |
| id | d803facd-623b-42ba-9620-372f4815b513 |
| name | api-prod.testdu.uib.no. |
| project_id | 6735f97c792c4bb3b342b920b1f13f02 |
| records | 158.37.65.111 |
| | 158.37.65.7 |
| | 158.37.65.60 |
| status | PENDING |
| ttl | None |
| type | A |
| updated_at | None |
| version | 1 |
| zone_id | f2c1d504-d3cc-4d3c-a602-1940688f4b81 |
| zone_name | testdu.uib.no. |
+-------------+--------------------------------------+
For real production, we have created a domain for the production use of Hono API: api.ub.uib.no
For Rancher production deployed in Microk8s cluster, we use NREC project uib-ub-kubernetes-rancher in BGO region, and 3 instances (Rocky Linux 9) have been created for Microk8s Kubernetes cluster:
kubernetes-rancher-prod-N01: 158.39.201.151
kubernetes-rancher-prod-N02: 158.39.77.227
kubernetes-rancher-prod-N03: 158.37.65.37
There are in total 6 volumes created.
Three volumes are mounted to the Microk8s directory /var/snap:
Name | Size | Type | Attached To
--------------------------------------------------------------------
microk8s-vol-prod-N01 - 50GiB - rbd - /dev/sdb on kubernetes-rancher-prod-N01
microk8s-vol-prod-N02 - 50GiB - rbd - /dev/sdb on kubernetes-rancher-prod-N02
microk8s-vol-prod-N03 - 50GiB - rbd - /dev/sdb on kubernetes-rancher-prod-N03
Other volumes for the MicroCeph (if needed, but currently we have not set MicroCeph cluster to connect Microk8s cluster for these 3 instances):
Name | Size | Type | Attached To
--------------------------------------------------------------------
microceph-vol-prod-N01 - 50GiB - rbd - /dev/sdc on kubernetes-rancher-prod-N01
microceph-vol-prod-N02 - 50GiB - rbd - /dev/sdc on kubernetes-rancher-prod-N02
microceph-vol-prod-N03 - 50GiB - rbd - /dev/sdc on kubernetes-rancher-prod-N03
The current domain for rancher deployed to Microk8s Kubernetes cluster is rancher-prod.testdu.uib.no.
We can check it by OpenStack command: openstack recordset list testdu.uib.no.:
It is created by OpenStack command:
openstack recordset create testdu.uib.no. rancher-prod --type A --record 158.39.201.151 --record 158.39.77.227 --record 158.37.65.37
The domain DNS A record points to all three IPs of NREC instance nodes to achieve round-robin access to instances. Then, the Ingress of the Microk8s Kubernetes cluster handles the access to the Rancher through its clusterIP:
[rocky@kubernetes-rancher-prod-n01 ~]$ kubectl get ingress rancher-ingress -n cattle-system
NAME CLASS HOSTS ADDRESS PORTS AGE
rancher-ingress public rancher-prod.testdu.uib.no 127.0.0.1 80, 443 76d
[rocky@kubernetes-rancher-prod-n01 ~]$ kubectl get svc -n cattle-system -l app=rancher
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
rancher ClusterIP 10.152.183.126 <none> 80/TCP,443/TCP 76d
Here is the template rancher-ingress.yml implemented in microk8s-NREC-deployment repo, which is used to set up ingress by Ansible:
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: rancher-ingress
namespace: cattle-system
annotations:
cert-manager.io/cluster-issuer: lets-encrypt
spec:
tls:
- hosts:
- "{{ rancher_domain }}"
secretName: rancher-ingress-tls
rules:
- host: "{{ rancher_domain }}"
http:
paths:
- pathType: Prefix
path: "/"
backend:
service:
name: rancher
port:
number: 80
and the ansible code to execute the template is implemented in microk8s-NREC-deployment repo:
- name: Create ingress for rancher based on Lets-encrypt
k8s:
kubeconfig: /var/snap/microk8s/current/credentials/client.config
state: present
definition: "{{ lookup('template', 'rancher-ingress.yml') | from_yaml }}"