IT: Kolla Ansible OpenStack Deployment - feralcoder/shared GitHub Wiki

Up Links

Public

feralcoder public Home feralcoder IT Living Room Data Center

My Private Cloud

I'm building on enterprise servers and network, with an entirely production mindset. As much as is possible in my living room. I've hijacked 240V power from the kitchen stove for this, with redundant power from 2 other house circuits.

HOWTOS

HOWTO: Install Kolla-Ansible HOWTO: Install Ceph HOWTO: Setup Docker Registries For OpenStack HOWTO: Kolla-Ansible Container Management HOWTO: Setup Octavia LBAAS

History

Abandoned: OpenStack On OpenStack

I started with OOO before turning to Kolla-Ansible to manage my deployment. OOO uses a template system to drive Puppet, to drive Ansible, to build an Undercloud, to build the Overcloud. There are compelling reasons to want to drive via templates: this system can be built in many ways, for many needs, with a multitude of components and options. And it's appealing to build a minimal single-host Undercloud, then leverage that to bootstrap the full OpenStack Overcloud.

But the problems were too many, the upstream breakages came too frequently, and the debugging was too complicated. These weren't 'Cliff Problems', because frequently I'd lose days hacking around something only to discover that problem resolved a week later in the upstream code. I discovered simple problems were obfuscated by the layers on layers in front of them: Ansible, Puppet, and OOO's templating and command output management.

As the Kolla-Ansible project members point out, and I wholeheartedly agree in this case, it's folly to try to simplify deployment of a complicated error-prone system of systems by using that same system to deploy itself.

I learned a bit about OpenStack in the months I spent working with OOO. But most of my lessons were around how to work with OOO itself - worthless experience after abandoning this approach.

Kolla-Ansible Is The Way

I spent months working through OOO without ever deploying a working stack. I spent days on Kolla-Ansible before I had a working stack. It's not perfect: it's just as dynamic as the OOO project, with upstream breakages and documentation gaps, but resolving these is so much more straightforward.

I continue to layer increasing complexity, so I continue rebuilding and working through issues. This blog post captures a point-in-time that is nearly production-worthy. Most recently I've layered on Octavia LBAAS and Magnum container orchestration, both of which deserve their own pages.

Kolla is both a container lifecycle management system, and a community building tools around that system. Generally, Kolla takes upstream OpenStack code and builds Docker images from that. You can use Kolla-Ansible to pull these built containers to your system, or you can use Kolla to build custom versions for yourself. You can then use Kolla-Ansible to manage the lifecycle of your stack, and use Kolla directly for deeper control of container configurations.

The Future: More Kolla

Currently I'm building my server and network environment before starting the Kolla-Ansible deployment. The larger Kolla project also maintains Koyobe (Kolla-on-Bifrost). Bifrost is a baremetal and network provisioning system, and given my good experience with Kolla-Ansible, I expect Koyobe to be a good way to provision from scratch.

Current State of Success and Failure

My Ceph cluster is redundant, top to bottom, and secure:

  • 3 Manager / Monitor nodes for a HA control plane

    • radosgw object API, serving images
    • cephx auth required within cluster
    • cephx auth required of clients
    • cephfs enabled, metadata layer for future file store clients
  • 5 OSD

My OpenStack deployment currently includes:

  • Core Components
    • Octavia
    • Magnum
    • Manila

Versions

Despite reasons to want to move to Wallaby (ie Manila issues) I'm locked into OpenStack:Victoria on Ceph:Nautilus for now... Nope. OpenStack:Wallaby on Ceph:Nautilus for now, with hacks. IT: My Cloud Versions

My Infrastructure

Ceph

Kolla-Ansible used to also deploy Ceph. This is no longer the case. I'm using ceph-ansible to deploy ceph. There are some mismatced configurations to resolve before this layering works.

See the HOWTO: Install Ceph page for details on installing Ceph.

Network

I'm using a NIC per network to achieve isolation. Since my servers have differently named interfaces (combination of dl380p G8's, bl460c G6's, and bl460c G8's), I've created consistently named bonds on every server. Each bond has only one NIC in it, but it gives me a single naming scheme to deploy onto.

See the HOWTO: Install Kolla-Ansible page for details on setting up the network.

Local Docker Registries

It's highly advisable to set up a local Docker pull-thru registry, so you're only pulling each container once, however many hosts you deploy to, and however many times you redeploy.

I've got reasons to build my own Kolla containers, so I need a local Docker registry to push them to. Even without custom containers, you'd benefit from setting up a local Docker Registry to push to:

  • Kolla containers are only versioned by OpenStack release, ie 'victoria', 'ussuri', etc.
  • Kolla maintainers push updates daily.
  • Kolla-Ansible plus Docker has no mechanism to pull what's in the pull-thru registry without checking upstream for updates.

To achieve "version-pinning", I do something like this:

  • Use kolla-ansible to pull the default kolla/*:victoria images
    • This can be configured to automatically pull source and build local containers, too.
  • I iterate over all images in my pull-thru cache:
  • I reconfigure kolla-ansible to use my local registry with my pinned version
    • Build
    • Build
    • Build

Since you can't push packages to a pull-thru registry, you'll need 2 Docker registries to do version pinning.

Custom Containers

I build custom containers, for a few reasons:

  • Binary containers don't always come ready to support other optional components.
    • The horizon dashboard isn't built with templates to support the magnum container manager, leaving you with CLI but no GUI control on your Kubernetes clusters.
  • It's nice to have the code local to investigate things.
  • I still don't have Octavia running, and I believe the fix will involve some customization of the Octavia container.

Docker Registry Setup

See the HOWTO: Setup Docker Registries page for details on setting up local and pull-thru Docker registries.