feralcoder public Home
feralcoder IT
Living Room Data Center

My Private Cloud
Kolla-Ansible OpenStack Deployment

HOWTO: Install Kolla-Ansible
HOWTO: Install Ceph
HOWTO: Kolla-Ansible Container Management
HOWTO: Setup Octavia LBAAS

My Ceph and OpenStack deployment process requires 2 different registry setups:

• Pull-thru Registry, to cache all the upstream containers I use over many hosts and deployments
• Local Registry, which I can push retagged and custom containers to

The local registry allows me to pin container versions where upstream versioning doesn't allow. For example, Kolla-Ansible containers are versioned only as "Victoria". I retag these with datestamps and push them locally so that my deployments are predictable and locally sourced.

See IT: Kolla Ansible-OpenStack Deployment for more discussion on the need for local registries.

See IT: HOWTO: Kolla-Ansible Container Management for how to use these registries to manage the container lifecycle.

sudo dnf -y install yum-utils
sudo yum-config-manager     --add-repo     https://download.docker.com/linux/centos/docker-ce.repo
sudo dnf -y install docker-ce
sudo mkdir /etc/docker

Place /etc/docker/daemon.json:

{
    "bip": "172.26.0.2/16",
    "insecure-registries": [
        "192.168.127.220:4001"
    ],
    "log-driver": "journald",
    "log-opts": {
        "tag": "{{.Name}}"
    },
    "registry-mirrors": [
        "http://192.168.127.220:4000"
    ]
}

I redefine the base IP range because the default conflicts with subnets I'm using in my OpenStack cluster.

I serve the pull-thru cache on port 4000, and my local registry on 4001. I also prefer to log to journald, modify this to your own preference.

Now, start Docker.

sudo systemctl enable --now docker

You'll want the registries to store onto a volume outside the containers. This will allow you to locate container storage separate from registry storage, and also more easily manage registry persistence across rebuilds. As the registry containers are destroyed and recreated, all registry state will persist in the cache.

I mount a /registry/ volume from a raided array which persists acrross host rebuilds.

This line in the registry start scripts does the mapping:

 -v /registry/docker/local-registry:/var/lib/registry

sudo vi /usr/local/bin/docker-local-registry-start.sh

#!/bin/bash

docker run -d \
 --name docker-local-registry \
 --restart=always \
 -p 4001:5000 \
 -v /registry/docker/local-registry:/var/lib/registry \
 registry:2

sudo vi /usr/local/bin/docker-local-registry-stop.sh

#!/bin/bash

CONTAINER_ID=`docker container list -a | grep 'docker-local-registry' | awk '{print $1}'`
docker container stop $CONTAINER_ID
docker container rm $CONTAINER_ID

sudo vi /etc/systemd/system/docker-local-registry.service

[Unit]
Description=Docker Local Registry Daemon
After=docker.service
Requires=docker.service


[Service]
Type=oneshot
User=root
RemainAfterExit=yes
ExecStartPre=-/usr/local/bin/docker-local-registry-stop.sh
ExecStart=/usr/local/bin/docker-local-registry-start.sh
ExecStop=/usr/local/bin/docker-local-registry-stop.sh
[Install]
WantedBy=multi-user.target

sudo mkdir -p /registry/docker-local-registry
sudo chmod 755 /usr/local/bin/docker-local-registry-start.sh
sudo chmod 755 /usr/local/bin/docker-local-registry-stop.sh
sudo chmod 644 /etc/systemd/system/docker-local-registry.service

sudo systemctl enable docker-local-registry
sudo systemctl start docker-local-registry

sudo vi usr/local/bin/docker-pullthru-registry-start.sh

#!/bin/bash

docker run -d \
 --name docker-pullthru-registry \
 --restart=always \
 -p 4000:5000 \
 -v /registry/docker/pullthru-registry:/var/lib/registry \
 -e REGISTRY_PROXY_REMOTEURL=https://registry-1.docker.io \
 registry:2

sudo vi /usr/local/bin/docker-pullthru-registry-stop.sh

#!/bin/bash

CONTAINER_ID=`docker container list -a | grep 'docker-pullthru-registry' | awk '{print $1}'`
docker container stop $CONTAINER_ID
docker container rm $CONTAINER_ID

sudo vi /etc/systemd/system/docker-pullthru-registry.service

[Unit]
Description=Docker PullThru Registry Daemon
After=docker.service
Requires=docker.service


[Service]
Type=oneshot
User=root
RemainAfterExit=yes
ExecStartPre=-/usr/local/bin/docker-pullthru-registry-stop.sh
ExecStart=/usr/local/bin/docker-pullthru-registry-start.sh
ExecStop=/usr/local/bin/docker-pullthru-registry-stop.sh
[Install]
WantedBy=multi-user.target

sudo mkdir -p /registry/docker-pullthru-registry
sudo chmod 755 /usr/local/bin/docker-pullthru-registry-start.sh
sudo chmod 755 /usr/local/bin/docker-pullthru-registry-stop.sh
sudo chmod 644 /etc/systemd/system/docker-pullthru-registry.service

sudo systemctl enable docker-pullthru-registry
sudo systemctl start docker-pullthru-registry

sudo firewall-cmd --zone=public --permanent --add-service=4000
sudo firewall-cmd --zone=public --permanent --add-service=4001
sudo firewall-cmd --reload

All clients should have the same daemon.conf installed as above, then restart Docker.

sudo systemctl restart Docker

Run tcpdump on the registry server and pull a container image on a client to verify Docker is behaving properly.

On registry server:

sudo dnf -y install tcpdump
sudo tcpdump -i any port 4000 or port 4001 or port 5000 or port 443  -nn

On a client run:

sudo docker pull centos

Client IP: 192.168.127.214
Registry Server IP: 192.168.127.220
Pull-Thru Cache IP: 172.26.0.3

You should see something like:

# Client initiates connection to RegistryServer:4000:
01:39:50.277692 IP 192.168.127.214.38278 > 192.168.127.220.4000: Flags [S], seq 690959748, win 29200, options [mss 1460,sackOK,TS val 3863841244 ecr 0,nop,wscale 7], length 0

# RegistryServer relays that to PullThruCacheContainer:5000:
01:39:50.277774 IP 192.168.127.214.38278 > 172.26.0.3.5000: Flags [S], seq 690959748, win 29200, options [mss 1460,sackOK,TS val 3863841244 ecr 0,nop,wscale 7], length 0

# PullThruCacheContainer replies to Client:
01:39:50.277844 IP 172.26.0.3.5000 > 192.168.127.214.38278: Flags [S.], seq 1469235457, ack 690959749, win 28960, options [mss 1460,sackOK,TS val 1130058506 ecr 3863841244,nop,w scale 7], length 0

# Reply is translated to RegistryServer's source IP:
01:39:50.277859 IP 192.168.127.220.4000 > 192.168.127.214.38278: Flags [S.], seq 1469235457, ack 690959749, win 28960, options [mss 1460,sackOK,TS val 1130058506 ecr 3863841244, nop,wscale 7], length 0

# Client makes request to RegistryServer:
01:39:50.278419 IP 192.168.127.214.38278 > 192.168.127.220.4000: Flags [P.], seq 1:256, ack 1, win 229, options [nop,nop,TS val 3863841245 ecr 1130058506], length 255

# PullThruCacheContainer makes requests to internet, Source IPs translated to RegistryServer IP:
01:39:53.144461 IP 172.26.0.3.38842 > 52.5.11.128.443: Flags [P.], seq 2745:5141, ack 5798, win 342, options [nop,nop,TS val 2793310011 ecr 845728410], length 2396
01:39:53.144482 IP 192.168.127.220.38842 > 52.5.11.128.443: Flags [P.], seq 2745:5141, ack 5798, win 342, options [nop,nop,TS val 2793310011 ecr 845728410], length 2396
01:39:53.144925 IP 172.26.0.3.53430 > 54.85.56.253.443: Flags [P.], seq 314:519, ack 5326, win 319, options [nop,nop,TS val 2734574072 ecr 3719506185], length 205
01:39:53.144941 IP 192.168.127.220.53430 > 54.85.56.253.443: Flags [P.], seq 314:519, ack 5326, win 319, options [nop,nop,TS val 2734574072 ecr 3719506185], length 205
01:39:53.150305 IP 172.26.0.3.39580 > 18.213.137.78.443: Flags [S], seq 3847216542, win 29200, options [mss 1460,sackOK,TS val 2921814609 ecr 0,nop,wscale 7], length 0
01:39:53.150325 IP 192.168.127.220.39580 > 18.213.137.78.443: Flags [S], seq 3847216542, win 29200, options [mss 1460,sackOK,TS val 2921814609 ecr 0,nop,wscale 7], length 0

# Internet IPs reply to RegistryServer IP, destination translated to PullThruContainer IP:
01:39:53.159406 IP 52.5.11.128.443 > 192.168.127.220.38852: Flags [S.], seq 4156388022, ack 3930161699, win 26847, options [mss 1460,sackOK,TS val 845728425 ecr 2793309926,nop,wscale 8], length 0
01:39:53.159433 IP 52.5.11.128.443 > 172.26.0.3.38852: Flags [S.], seq 4156388022, ack 3930161699, win 26847, options [mss 1460,sackOK,TS val 845728425 ecr 2793309926,nop,wscale 8], length 0
...