Configure OSA Multi node Environment - nuagenetworks/nuage-openstack-ansible GitHub Wiki

Introduction

Please refer to the OSA Install guide for detailed instructions to setup a multi-node OpenStack environment

The minimum deployment recommended is as follows:

  • Three infrastructure (a.k.a. controller) nodes
  • One logging node
  • One compute node

One of the infrastructure hosts (infra1) doubled as the deployment host. The deployment host is where the openstack-ansible code is cloned, ansible is installed, and the playbooks are run to accomplish the deployment.

Configuration

VM configuration

All five nodes used in the deployment are Ubuntu 14.04 Server VMs running under KVM on a bare-metal host that also was running Ubuntu 14.04 Server. OSA deployments in production would generally occur on bare metal hosts rather than VMs but this document refers to a complete VM based deployment. KVM was configured for the following virtual neworks:

  • "data" network, NAT-enabled, DHCP-enabled, addresses 192.168.1.200-254 set aside for static IPs
  • "system" network, no NAT, no DHCP

Each VM was configured to have the following resources:

  • 100 GB Disk
  • 4096 MB RAM
  • 2 CPU
  • 2 vNICs
  • eth0 was on the "data" network
  • eth1 was on the "system" network

Node Network Configuration

The openstack-ansible installation document describes a general network configuration for each node. This configuration must be done manually on each node prior to playbooks being run. It is important to note, however, that the description in the installation guide assumes that the installer will be following best-practices using bonded interfaces. The deployment process detailed in this document assumes non-bonded interfaces.

A simplified diagram of the setup is shown, below:

Network Topology

Here is a sample /etc/network/interfaces file, taken from infra1:

# This file describes the network interfaces available on your system
# and how to activate them. For more information, see interfaces(5).
# The loopback network interface
auto lo
iface lo inet loopback

# The data network interface
auto eth0
iface eth0 inet static
    address 192.168.1.201
    gateway 192.168.1.1
    netmask 255.255.255.0
    dns-nameservers 192.168.1.1 128.251.10.29 128.251.10.125

# The system network interface
auto eth1
iface eth1 inet manual

# Container management VLAN interface
iface eth0.10 inet manual
    vlan-raw-device eth0

# OpenStack Networking VXLAN (tunnel/overlay) VLAN interface
iface eth1.30 inet manual
    vlan-raw-device eth1

# Storage network VLAN interface (optional)
iface eth0.20 inet manual
    vlan-raw-device eth0

# Container management bridge
auto br-mgmt
iface br-mgmt inet static
    bridge_stp off
    bridge_waitport 0
    bridge_fd 0
    offload-sg off
    # Bridge port references tagged interface
    bridge_ports eth0.10
    address 172.29.236.11
    netmask 255.255.252.0
    dns-nameservers 128.251.10.29 128.251.10.125

# OpenStack Networking VXLAN (tunnel/overlay) bridge
auto br-vxlan
iface br-vxlan inet static
    bridge_stp off
    bridge_waitport 0
    bridge_fd 0
    offload-sg off
    # Bridge port references tagged interface
    bridge_ports eth1.30
    address 172.29.240.11
    netmask 255.255.252.0

# OpenStack Networking VLAN bridge
auto br-vlan
iface br-vlan inet manual
    bridge_stp off
    bridge_waitport 0
    bridge_fd 0
    offload-sg off
    # Bridge port references untagged interface
    bridge_ports eth1

# Storage bridge (optional)
auto br-storage
iface br-storage inet static
    bridge_stp off
    bridge_waitport 0
    bridge_fd 0
    offload-sg off
    # Bridge port reference tagged interface
    bridge_ports eth0.20
    address 172.29.244.11
    netmask 255.255.252.0 

user_secrets.yml

By default, the file /etc/openstack_deploy/user_secrets.yml has no passwords. If this file isn't updated, OSA deployment will eventually fail due to a lack of permissions. This can be avoided by following the instructions: http://docs.openstack.org/developer/openstack-ansible/liberty/install-guide/configure-creds.html

Load Balancer, haproxy, and repo servers

Each node will use its Container Management bridge network during the deployment for various operations. For example, when a container on a node wants to do a pip install, it tries to contact the pypi server using the IP address that you define for internal_lb_vip_address in the /etc/openstack_deploy/openstack_user_config.yml file. In absence of a real load balancer IP, use the container management IP address of the haproxy server. In this document, infra1 serves as the haproxy server.

Now, for the haproxy server to work as a load balancer in this pip case, designate the infrastructure nodes as repo nodes. This will allow haproxy to be configured to forward pip requests to one of the infrastructure hosts as server.

To accomplish these things, ensure that the following parameters are configured in the /etc/openstack_deploy/openstack_user_config.yml file. Note that this is not the complete file. Snippets are shown for reference:

#
# Define three shared infrastructure hosts:
#
shared-infra_hosts:
  infra1:
    ip: 172.29.236.11
  infra2:
    ip: 172.29.236.12
  infra3:
    ip: 172.29.236.13
#
# Define haproxy host:
#
haproxy_hosts:
  infra1:
    ip: 172.29.236.11
#
# Define three package repository hosts:
#
repo-infra_hosts:
  infra1:
    ip: 172.29.236.11
  infra2:
    ip: 172.29.236.12
  infra3:
    ip: 172.29.236.13
#
# Define global overrides
#
global_overrides:
  internal_lb_vip_address: 172.29.236.11
  external_lb_vip_address: 172.29.236.11

Logging

To enable logging during an ansible playbook run, edit /opt/openstack-ansible/playbooks/ansible.cfg and add a single line. For example:

log_path = /opt/openstack-ansible/playbooks/ansible.log

Verbose

The openstack-ansible command accommodates different levels of verbosity. If you append '-v', '-vv', '-vvv', or '-vvvv' to the command to run a playbook you get increasing levels of verbosity in the output and in the logs. '-vvvv' is the highest level and includes connectivity information.

Failure

When a failure running a playbook occurs, ansible deposits a file in your ~ folder. This file can be used on the next playbook run to limit what ansible will redo. If used, openstack-ansible won't redo some operations that had completed successfully. The syntax for using this requires that you add something like the following to your command line:

--limit @/home/ubuntu/setup-hosts.retry

The example, above, is following a failure while running the setup-hosts.yml playbook.

Note that some playbooks are nothing more than execution of a list of other playbooks. setup-hosts, for example, executes:

  • openstack-hosts-setup.yml
  • lxc-hosts-setup.yml
  • lxc-containers-create.yml

This example is illustrative of a couple of things that can be useful during debugging.

First, to reduce the time required to re-try things, you can simply re-execute the sub-playbook where the failure occurred instead of running all 3.

Second, some playbooks have compliments. If you look on disk, you can find both lxc-containers-create.yml and lxc-containers-destroy.yml. You can "start over" by running destroy and re-running create.

--limit

--limit is a command line option that can be used to reduce the scope of the execution of a playbook. Using the "destroy" example, above, one can limit the scope of lxc-containers-destroy.yml with the following example:

openstack-ansible lxc-containers-destroy.yml --limit keystone_all

This example would destroy all the keystone containers, leaving all other containers intact.