feralcoder public Home
feralcoder IT
Living Room Data Center

My Private Cloud
Kolla-Ansible OpenStack Deployment

HOWTO: Install Kolla-Ansible
HOWTO: Install Ceph
HOWTO: Setup Docker Registries For OpenStack
HOWTO: Setup Octavia LBAAS

I'm building OpenStack:Victoria via Kolla-Ansible, with Ceph:Nautilus via Ceph-Ansible, on CentOS 8. Both deployments are containerized on baremetal. I'm using Docker instead of Podman, for reasons in the kolla-ansible code, and am having to fight code in ceph-ansible to do it, more on that in the Ceph HOWTO linked above.

You can use containers built from binary packages or source, configured via kolla_install_type=[binary|source] in globals.yml. This setting does not mean "fetch built containers" vs "pull code and build". The difference is, rather, "containers built from RPM's" vs "containers built from source". Docker.io hosts both types, and you can build both types locally.

The kolla-ansible images are only tagged by OpenStack release upstream, eg "victoria". As published, there's no way for me to guarantee version stability across rebuilds. Even when I use a local pull-thru cache, there's no way to guarantee my next build will be using a version in that cache.

To give myself version stability I have separated the image pull process from deployment, and I retag all images to a point-in-time identifier. I then push those localized images into a local Docker registry. This allows me to deploy predictably and quickly, and update containers for future building and testing outside the deployment cycle. Also, if I run into issues after an update, I have the earlier version tagged locally to roll back to.

A local registry is also required to customize and build my own container versions.

Since you can't push packages to a pull-thru registry, I need 2 Docker registries to do this cacheing and versioning.

My fetching (of prebuilt images) is still done by the kolla-ansible scripts, be it updates from upstream, or pinned localized versions. I toggle settings in /etc/kolla/globals.yml at different stages of deployment to do the fetch-localize-push-use dance.

• When I run "kolla-ansible bootstrap-servers" I need the local registry configured, so that stack hosts are configured with appropriate insecure-registries Docker configs.
• I use "kolla-ansible pull" to prefetch the latest packages, when I want to update. For this I reconfigure globals.yml to point at "kolla/*:victoria" images."
• After I fetch the latest containers, I run a loop on one of my stack hosts to pull them from my pull-through cache, tag them to my local registry with a date stamp tag, and push them to my local registry.
• Before I run the actual deploy I configure globals.yml to use my local registry and tags.

These are the settings of interest in /etc/kolla/globals.yml:

## PINNED CONTAINER VERSIONS
#docker_registry: 192.168.127.220:4001
#docker_namespace: "feralcoder"
#openstack_release: "feralcoder-20210321"
#kolla_install_type: "binary"

# LATEST CONTAINER VERSIONS
docker_registry:
docker_registry_username: feralcoder
docker_namespace: "kolla"
openstack_release: "victoria"
kolla_install_type: "binary"

My pseudocode looks something like this:

INSTALL_TYPE=binary

use_localized_containers () {
  cp $KOLLA_SETUP_DIR/files/kolla-globals-localpull.yml /etc/kolla/globals.yml
  cat $KOLLA_SETUP_DIR/files/kolla-globals-remainder.yml >> /etc/kolla/globals.yml
}

use_latest_dockerhub_containers () {
  # We switch to dockerhub container fetches, to get the latest "victoria" containers
  cp $KOLLA_SETUP_DIR/files/kolla-globals-dockerpull.yml /etc/kolla/globals.yml
  cat $KOLLA_SETUP_DIR/files/kolla-globals-remainder.yml >> /etc/kolla/globals.yml
}

localize_latest_containers () {
  KOLLA_PULL_THRU_CACHE=/registry/docker/pullthru-registry/docker/registry/v2/repositories/kolla/
  for CONTAINER in `ls -d $KOLLA_PULL_THRU_CACHE/*${INSTALL_TYPE}*`; do
    ssh root@$PULL_HOST "docker image pull kolla/$CONTAINER:victoria"
    ssh root@$PULL_HOST "docker image tag kolla/$CONTAINER:victoria $LOCAL_REGISTRY/feralcoder/$CONTAINER:$TAG"
    ssh root@$PULL_HOST "docker image push $LOCAL_REGISTRY/feralcoder/$CONTAINER:$TAG"
  done
}

use_localized_containers 
kolla-ansible -i $INVENTORY  bootstrap-servers
use_latest_dockerhub_containers
kolla-ansible -i $INVENTORY pull
localize_latest_containers
use_localized_containers
kolla-ansible -i $INVENTORY deploy

To build your own containers, you'll need to use Kolla, the container lifecycle system which Kolla-Ansible is built around.

When building your own containers you'll benefit by having local RPM/APT repos so all your containers aren't refetching all the same packages over the internet. It's essential, really. I won't explain this process here, it's straightforward and well documented by the internet. There are many ways to do it, and your solution will depend on your own preferences and environment, but it should be easy. The hardest part will be waiting the day(s) for your initial syncs to finish.

After you've set up a local package repo, you'll need to modify the base container image to point at those repos. Then you can build. I will go over this part.

Setup the environment:

sudo yum -y install epel-release
sudo yum -y install python36-devel git python3 python3-pip gcc
pip3 install --upgrade pip
mkdir -p ~/CODE/openstack  &&  cd ~/CODE/openstack
git clone https://github.com/openstack/kolla.git
cd kolla
git checkout stable/victoria
cd ..
pip3 install kolla/
pip3 install tox

cd kolla
tox -e genconfig

fetch_kolla_container_source () {
  cd $KOLLA_CODE_DIR                                                          || return 1
  cat $SUDO_PASS_FILE | sudo -S ls > /dev/null                                || return 1
  sudo mkdir -p $NOW_TARBALLS && sudo chown cliff:cliff $NOW_TARBALLS         || return 1
  grep '^#location = .*tar.gz' etc/kolla/kolla-build.conf > $NOW_TARBALLS/locations || return 1
  sed -i 's|^#location = .tarballs_base|wget -P $NOW_TARBALLS https://tarballs.opendev.org|g' $NOW_TARBALLS/locations || return 1
  sed -i 's|^#location = |wget -P $NOW_TARBALLS |g' $NOW_TARBALLS/locations   || return 1
  sed -E -i 's/^(wget .*)/\1 || return 1/g' $NOW_TARBALLS/locations           || return 1
  . $NOW_TARBALLS/locations
}

build_kolla_containers () {
  cd $KOLLA_CODE_DIR   || return 1
  cat etc/kolla/kolla-build.conf | sed -E 's/#type = url/type = local/g' |  sed -E "s|^#location = .tarballs_base.*/([^/]*.tar.gz)|location = $NOW_TARBALLS/\1|g" | sed -E "s|^#location = .*/([^/]*.tar.gz)|location = $NOW_TARBALLS/\1|g" > etc/kolla/kolla-build-local.conf
  # kolla will use tag "8" with following base image...
  BASE_IMAGE="--base-image $LOCAL_DOCKER_REGISTRY/feralcoder/centos-feralcoder"
  kolla-build -t source -b centos $BASE_IMAGE --push --registry $LOCAL_DOCKER_REGISTRY -n feralcoder --tag $TAG   --config-file etc/kolla/kolla-build-local.conf || return 1
}

tag_as_latest () {
  for CONTAINER in `docker image list | grep $TAG | awk '{print $1}'`; do
    docker tag $CONTAINER:$TAG $CONTAINER:latest    || return 1
    docker push $CONTAINER:latest                   || return 1
  done
}

Set your image preferences in /etc/kolla/globals.yml:

# Valid options are [ binary, source ]
kolla_install_type: "source"

# Valid options are ['centos', 'debian', 'rhel', 'ubuntu']
kolla_base_distro: "centos"

Kick off an image pull, as above, and wait for the builds to happen...

xxx

Iterate over

It's highly advisable to set up a local Docker pull-thru registry, so you're only pulling each container once, however many hosts you deploy to, and however many times you redeploy.

I've got reasons to build my own Kolla containers, so I need a local Docker registry to push them to. Even without custom containers, you'd benefit from setting up a local Docker Registry to push to:

• Kolla containers are only versioned by OpenStack release, ie 'victoria', 'ussuri', etc.
• Kolla maintainers push updates daily.
• Kolla-Ansible plus Docker has no mechanism to pull what's in the pull-thru registry without checking upstream for updates.

To achieve "version-pinning", I do something like this:

• Use kolla-ansible to pull the default kolla/*:victoria images
  • This can be configured to automatically pull source and build local containers, too.
• I iterate over all images in my pull-thru cache:
  • Retag each to my local repo, with a datestamped tag, eg: http://192.168.127.220/feralcoder/centos-binary-haproxy/feralcoder-20210403
  • Push to my local registry
• I reconfigure kolla-ansible to use my local registry with my pinned version
  • Build
  • Build
  • Build

I build custom containers, for a few reasons:

• Binary containers don't always come ready to support other optional components.
  • The horizon dashboard isn't built with templates to support the magnum container manager, leaving you with CLI but no GUI control on your Kubernetes clusters.
• It's nice to have the code local to investigate things.
• I still don't have Octavia running, and I believe the fix will involve some customization of the Octavia container.