Issues connecting to Private "insecure registries" in containerd docker - q-uest/notes-doc-k8s-docker-jenkins-all-else GitHub Wiki

Some of the errors/issues encountered for not having the proper setup for containerd/docker:

Failed to pull image "10.182.0.16:8082/appointme-admin-api:34": rpc error: code = Unknown desc = failed to pull and unpack image "10.182.0.16:8082/appointme-admin-api:34": failed to resolve reference "10.182.0.16:8082/appointme-admin-api:34": failed to do request: Head "https://10.182.0.16:8082/v2/appointme-admin-api/manifests/34": http: server gave HTTP response to HTTPS client

Failed to pull image "10.182.0.16:8082/appointme-admin-api:33": rpc error: code = Unknown desc = failed to pull and unpack image "10.182.0.16:8082/appointme-admin-api:33": failed to resolve reference "10.182.0.16:8082/appointme-admin-api:33": pulling from host 10.182.0.16:8082 failed with status code [manifests 33]: 401 Unauthorized

Check the below, if any of the above errors were received to fix them [IT DOES NOT HAVE TO BE EXECUTED ON ALL THE NODES OF A K8S CLUSTER; RUNNING THE STEP ON MASTER NODE WILL BE ENOUGH; THE SECRET OBJECT IN K8S WILL BE USED BY THE NODES WHICH TAKES CARE OF THE AUTHENTICATION PART OF IT ]:

1)Make sure the private repository is connectable from "docker"

docker login 10.182.0.16:8082  -u admin -p admin

If the above one is not working or connectable, check the the registry connect string provided for "insecure-registries" as below in /etc/docker/daemon.json.

{ "insecure-registries" : ["10.182.0.16:8082"] }

Did you make any of the changes above, Restart the Docker service before retrying anything.

systemctl restart docker

The below entries should be there in the "/etc/containerd/config.toml".

[plugins.cri.registry.mirrors]
[plugins.cri.registry.mirrors."10.182.0.16:8082"]
  endpoint = ["http://10.182.0.16:8082"]

Restart the Containerd service., if you make any changes to the above config file.

Create kubernetes secret object & pass it on as "imagePullSecrets" in your deployment/pod manifest files.

kubectl create secret generic regcred     --from-file=.dockerconfigjson=/root/.docker/config.json     --type=kubernetes.io/dockerconfigjson

===

[Unknown Authority : ](x509: certificate signed by unknown authority)

Error:

 Failed to pull image "10.182.0.3:8082/pcl:47-DEV": rpc error: code = Unknown desc = failed to pull and unpack image "10.182.0.3:8082/pcl:47-DEV": failed to resolve reference "10.182.0.3:8082/pcl:47-DEV": failed to do request: Head "https://10.182.0.3:8082/v2/pcl/manifests/47-DEV": x509: certificate signed by unknown authority

Configure the Docker Client Daemon to Trust the Certificate

Because we have configured HTTPs and we used a self-signed certificate, the docker client will not trust the certificate if we don’t configure it to do so.

For the docker engine to trust the self-signed certificate we used with NGINX/Nexus, we must place a copy of this certificate in a special directory on the client host. On the Docker clients (including all Kubernetes hosts), create a directory by the IP address of the Reveres proxy under /etc/docker/certs.d/ and copy the .crt (of the application, no certificate Authority file is required to be copied over) file generated earlier on the reverse proxy host.

This is how Docker client looks on Jenkins host (@/etc/docker/certs.d/35.202.196.238) with the certificates:

root@jenkins:/etc/docker/certs.d/34.67.194.116# ls -ltr
total 4
-rwxr-xr-x 1 root root 1712 May 29 16:55 app.crt

The same needs to be done on all the Docker/Kubernetes hosts.

Logging into Docker:

Provide the IP address of the Nginx host.

docker login -u admin -p password 34.67.194.116

The command adds /v2/ to the IP address automatically above and uses 8082 port as configured in Nginx above.

Try the below to see whether it works

docker pull 34.67.194.116/appointme-admin-api:129

Here, the IP address of the reverse proxy prefixed with the image works like an alias (with https) to the Nexus host. Behind the scenes, the configuration done at Nginx end does the redirection to the Nexus host.

Self-signed Certificate with GKE cluters & Docker Engine (Kubernetes versions < 1.19 only)

DO NOT need add the Reverse proxy's IP address/domain to "insecure-registries" in "/etc/docker/daemon.json".
Just Create Kubernetes Secret

In order to pull images through Kubernetes, create a kubernetes secret from the Dockerr's config.json as below. Otherwise, you may end up getting errors like ""unauthorized: authentication required" :

kubectl create secret generic regcred     --from-file=.dockerconfigjson=/root/.docker/config.json     --type=kubernetes.io/dockerconfigjson

Note:

You would see an entry having Nginx host's IP address @/root/.docker/config.json.

Self-signed Certificate with GKE cluters & Container.d (Kubernetes versions > 1.19)

Kubernetes version 1.19+ and COS/Ubuntu images based on containerd for GKE nodes. Before the 1.19 version Kubernetes used to use Docker for building images, but now it uses containerd. More details could be found in the official Google Cloud documentation.

Reference: https://stackoverflow.com/questions/67723381/how-to-put-self-signed-certificate-to-each-node-of-gke-cluster/67724696#67724696

The following commands to be executed on both master & slave nodes to copy/update the certificate:

sudo mkdir /usr/local/share/ca-certificates/extra
sudo cp  app.crt /usr/local/share/ca-certificates/extra/.
sudo update-ca-certificates
ls -l /etc/ssl/certs/app*
nsenter --target 1 --mount systemctl restart containerd

Note:

All the steps including the ones pertaining to "docker" also should have been executed without fail on ALL the nodes. It is also important to create and provide the K8s Secret object (created from the docker's config.json above) while creating deployments/pods. Otherwise, it will throw errors like the below:

Failed to pull image "34.125.112.64/appointme-admin-api:34": rpc error: code = Unknown desc = failed to pull and unpack image "34.125.112.64/appointme-admin-api:34": failed to resolve reference "34.125.112.64/appointme-admin-api:34": pulling from host 34.125.112.64 failed with status code [manifests 34]: 401 Unauthorized

apiVersion: apps/v1
kind: Deployment
metadata:
  name: nginx-deployment
spec:
  selector:
    matchLabels:
      app: nginx
  replicas: 2 # tells deployment to run 2 pods matching the template
  template:
    metadata:
      labels:
        app: nginx
    spec:
      containers:
      - name: nginx
        image: "34.125.87.205/appointme-admin-api:nexus-reg"
        ports:
        - containerPort: 80
      imagePullSecrets:
        - name: regcred