External route service access - stanislawbartkowski/CP4D GitHub Wiki

OpenShift routes

https://docs.openshift.com/container-platform/4.5/networking/routes/route-configuration.html

OpenShift route is an extension to Kubernetes service allowing external access to OpenShift/Kubernetes applications. But it requires at least one of the OpenShift nodes to be accessible directly from the client node.

Architecture

Assume the architecture where the whole OpenShift cluster is running on a private network and the gateway is a separate node running in both networks, private and public, but not being included in the OpenShift cluster.

Node Network Role
master0.oc.com private Master
worker0.oc.com private Worker
worker1.oc.com private Worker
worker2.oc.com private Worker
inf.oc.com private, public Gateway to the cluster, HAProxy, NFS services
client.sb.com public Client desktop

The client can access the gateway node and run OpenShift console or oc command utilizing the HAProxy gateway node service.

Create MySQL instance


This command creates MySQL instance and MySQL service.

oc new-app --as-deployment-config --docker-image=registry.access.redhat.com/rhscl/mysql-57-rhel7:latest --name=mysql-openshift -e MYSQL_USER=user1 -e MYSQL_PASSWORD=mypa55 -e MYSQL_DATABASE=testdb -e MYSQL_ROOT_PASSWORD=r00tpa55

oc get svc

NAME              TYPE        CLUSTER-IP     EXTERNAL-IP   PORT(S)    AGE
mysql-openshift   ClusterIP   172.30.94.36   <none>        3306/TCP   4d12h

Make sure that MySQL database is up and running.

oc get pods


NAME                      READY   STATUS    RESTARTS   AGE
mysql-openshift-1-85ft6   1/1     Running   1          26h

oc port-forward mysql-openshift-1-85ft6 3306:3306

Forwarding from 127.0.0.1:3306 -> 3306
Forwarding from [::1]:3306 -> 3306

Using another command-line console

mysql -uuser1 -pmypa55 --protocol tcp -h 127.0.0.1

mysql: [Warning] Using a password on the command line interface can be insecure.
Welcome to the MySQL monitor.  Commands end with ; or \g.
Your MySQL connection id is 3
Server version: 5.7.24 MySQL Community Server (GPL)

Copyright (c) 2000, 2020, Oracle and/or its affiliates. All rights reserved.

Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.

mysql>

Create a route

oc expose service mysql-openshift
oc get route

NAME              HOST/PORT                                       PATH   SERVICES          PORT       TERMINATION   WILDCARD
mysql-openshift   mysql-openshift-sb.apps.rumen.os.fyre.ibm.com          mysql-openshift   3306-tcp                 None

Route exposes mysql-openshift-sb.apps.rumen.os.fyre.ibm.com hostname and 3306 port. But hostname points to Gateway which is not part of the OpenShift cluster and Gateway is unable to forward the port to OpenShift route.

oc delete route mysql-openshift

Solution using the service

Discover ClusterIP

oc describe service mysql-openshift

Name:              mysql-openshift
Namespace:         sb
Labels:            app=mysql-openshift
                   app.kubernetes.io/component=mysql-openshift
                   app.kubernetes.io/instance=mysql-openshift
.....
Type:              ClusterIP
IP:                172.30.94.36
Port:              3306-tcp  3306/TCP
TargetPort:        3306/TCP
Endpoints:         10.254.12.23:3306
Session Affinity:  None

Make ClusterIP accessible from Gateway node

CluserIP address 172.30.94.36 is not visible from Gateway node because it is the part of internal OpenShift network.

On Gateway node

Step 1: Create IP route to reach ClusterIP (172.30.94.36) assuming OpenShift Master node 10.16.35.203

ip route add 172.30.94.0/24 via 10.16.35.203 dev eth0


Test *mysql-openshift* service (assuming MySQL client installed on Gateway node)

mysql -uuser1 -pmypa55 --protocol tcp -h 172.30.94.36

Welcome to the MariaDB monitor.  Commands end with ; or \g.
Your MySQL connection id is 8
Server version: 5.7.24 MySQL Community Server (GPL)

Copyright (c) 2000, 2018, Oracle, MariaDB Corporation Ab and others.

Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.

MySQL [(none)]>

Make IP route permanent to survive network restart.

vi /etc/sysconfig/network-scripts/route-eth0

ADDRESS0=172.30.94.0
NETMASK0=255.255.255.0
GATEWAY0=10.16.35.203

Configure HAProxy on OpenShift Infra node

Assuming public IP address of Gateway node is 9.30.220.176. Forward all traffic on 3306 port to ClusterIP.

vi /etc/haproxy/haproxy.cfg

listen mysql
        bind 9.30.220.176:3306
        mode tcp
        server server1 172.30.94.36:3306 check

Restart HAProxy.

systemctl restart haproxy

Test from the client desktop


mysql -uuser1 -pmypa55 --protocol tcp -h mysql-openshift-sb.apps.rumen.os.fyre.ibm.com

mysql: [Warning] Using a password on the command line interface can be insecure.
Welcome to the MySQL monitor.  Commands end with ; or \g.
Your MySQL connection id is 9
Server version: 5.7.24 MySQL Community Server (GPL)

Copyright (c) 2000, 2020, Oracle and/or its affiliates. All rights reserved.

Oracle is a registered trademark of Oracle Corporation and/or its
affiliates. Other names may be trademarks of their respective
owners.

Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.

mysql>

Use NodePort (MySQL)

Make sure that the service is using NodePort. Create service NodePort if only ClusterIP is used.

vi svcnode.yml

apiVersion: v1
kind: Service
metadata:
  name: mysql-node
  labels:
    name: mysql-node
spec:
  type: NodePort
  ports:
    - port: 3306
      nodePort: 30306
      name: mysql-tcp-node
  selector:
      deployment: mysql

oc get svc

mysql        ClusterIP   172.30.28.22    <none>        3306/TCP         100m
mysql-node   NodePort    172.30.5.161    <none>        3306:30306/TCP   66s

Modify HAProxy configuration.

vi /etc/haproxy/haproxy.cfg

frontend mysql-http
        bind *:3306
        default_backend mysql-http
        mode tcp
        option tcplog

backend mysql-http
        balance source
        mode tcp
        server worker0 10.17.118.48:30306 check
        server worker1 10.17.119.46:30306 check
        server worker2 10.17.127.79:30306 check

Reload HAProxy.

systemctl reload haproxy

Connect to MySQL server using HAProxy node hostname and 3306 port. HAProxy server redirects incoming 3306 request to 30306 port inside the cluster and mysql-node service is redirecting it again to 3306 port in corresponding MySQL pod.

mysql -h jobbery-inf -u<user> -p <database>

Use NodePort (PostgreSQL)

Make sure that service is exposing NodePort. Create NodePort service if it is not.

apiVersion: v1
kind: Service
metadata:
  name: postgresql-persistent-node
  labels:
    name: postgresql-persistent
spec:
  type: NodePort
  ports:
    - port: 5432
      name: 5432-tcp-node
  selector:
      deployment: postgresql-persistent

oc create -f <yaml file>

Because nodePort property is not defined, OpenShift will assign a random port.

oc get svc

NAME                         TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)          AGE
postgresql-persistent        ClusterIP   172.30.214.127   <none>        5432/TCP         17m
postgresql-persistent-node   NodePort    172.30.124.188   <none>        5432:31114/TCP   9m24s

Here port 31114 is assigned.

Modify haproxy configuration. In this test environment, only one master node exists. Add all master nodes for HA.

vi /etc/haproxy/haproxy.cfg

.............

frontend postgresql
        bind *:5432
        default_backend postgresql
        mode tcp
        option tcplog

backend postgresql
        balance source
        mode tcp
        server master0 10.26.4.13:31114 check

systemctl reload haproxy

Connect to PostgreSQL using standard 5432 port. HAproxy will redirect the connection to 31114 service port and the service will push the traffic to 5432 port PostgreSQL pod.

⚠️ **GitHub.com Fallback** ⚠️