Configure GKE cluster to use Self signed certificates with Container.d - q-uest/notes-doc-k8s-docker-jenkins-all-else GitHub Wiki

• Configure the Docker Client Daemon to Trust the Certificate

Because we have configured HTTPs and we used a self-signed certificate, the docker client will not trust the certificate if we don’t configure it to do so.

For the docker engine to trust the self-signed certificate we used for NGINX, we must place a copy of this certificate in a special directory on the client host. On the Docker clients (including all Kubernetes hosts), create a directory by the IP address of the Reveres proxy under /etc/docker/certs.d/ and copy the .crt (of the application, no certificate Authority file is required to be copied over) file generated earlier on the reverse proxy host.

This is how Docker client looks on Jenkins host (@/etc/docker/certs.d/35.202.196.238) with the certificates:

root@jenkins:/etc/docker/certs.d/34.67.194.116# ls -ltr
total 4
-rwxr-xr-x 1 root root 1712 May 29 16:55 app.crt

The same needs to be done on all the Docker/Kubernetes hosts.

• Logging into Docker:

Provide the IP address of the Nginx host.

docker login -u admin -p password 34.67.194.116

The command adds /v2/ to the IP address automatically above and uses 8082 port as configured in Nginx above.

Try the below to see whether it works

docker pull 34.67.194.116/appointme-admin-api:129

Here, the IP address of the reverse proxy prefixed with the image works like an alias (with https) to the Nexus host. Behind the scenes, the configuration done at Nginx end does the redirection to the Nexus host.

---

Perform the below step to make it work with Kubernetes versions < 1.19 only:

1. DO NOT need add the Reverse proxy's IP address/domain to "insecure-registries" in "/etc/docker/daemon.json".

2. Just Create Kubernetes Secret

In order to pull images through Kubernetes, create a kubernetes secret from the Dockerr's config.json as below. Otherwise, you may end up getting errors like ""unauthorized: authentication required" :

kubectl create secret generic regcred     --from-file=.dockerconfigjson=/root/.docker/config.json     --type=kubernetes.io/dockerconfigjson

Note:

You would see an entry having Nginx host's IP address @/root/.docker/config.json.

---

Kubernetes versions > 1.19 that are using "containerd":

Kubernetes version 1.19+ and COS/Ubuntu images based on containerd for GKE nodes. Before the 1.19 version Kubernetes used to use Docker for building images, but now it uses containerd. More details could be found in the official Google Cloud documentation.

Reference: https://stackoverflow.com/questions/67723381/how-to-put-self-signed-certificate-to-each-node-of-gke-cluster/67724696#67724696

The following commands to be executed on both master & slave nodes to copy/update the certificate:

sudo mkdir /usr/local/share/ca-certificates/extra
sudo cp  app.crt /usr/local/share/ca-certificates/extra/.
sudo update-ca-certificates
ls -l /etc/ssl/certs/app*
nsenter --target 1 --mount systemctl restart containerd

Note:

All the steps including the ones pertaining to "docker" also should have been executed without any fail on ALL the nodes. It is also important to create and provide the K8s Secret object (created from the docker's config.json above) while creating deployments/pods. Otherwise, it will throw errors like the below:

Failed to pull image "34.125.112.64/appointme-admin-api:34": rpc error: code = Unknown desc = failed to pull and unpack image "34.125.112.64/appointme-admin-api:34": failed to resolve reference "34.125.112.64/appointme-admin-api:34": pulling from host 34.125.112.64 failed with status code [manifests 34]: 401 Unauthorized

apiVersion: apps/v1
kind: Deployment
metadata:
  name: nginx-deployment
spec:
  selector:
    matchLabels:
      app: nginx
  replicas: 2 # tells deployment to run 2 pods matching the template
  template:
    metadata:
      labels:
        app: nginx
    spec:
      containers:
      - name: nginx
        image: "34.125.87.205/appointme-admin-api:nexus-reg"
        ports:
        - containerPort: 80
      imagePullSecrets:
        - name: regcred