Demonstration 2 - adjacentlink/python-etce-tutorial GitHub Wiki

Demonstration 2 builds on the single step test sequence from the previous demo:

1. We'll run the test on two LXC containers instead of the local host. To do so, we'll introduce etce-lxc and the ETCE LXC Plan File for creating networks of LXC containers and Linux bridges. The LXC Plan File leverages mako templates for compactly parameterizing configuration values that vary across containers.

2. Many Wrappers expose configuration arguments that can be set at run time. We'll use etce-wrapper to look at the utils.hello arguments and show how to set their values in steps.xml or with a separate ETCE Configuration File.

ETCE executes applications on test hosts over SSH. Whether the host is real, virtual, local or remote, if ETCE can connect to the target user account via public key authentication, it can use the host in tests.

The evolution of servers towards architectures containing dozens of CPUs means that ETCE can orchestrate relatively complex EMANE emulations on a single machine or two. Virtualization is the key to this and light-weight LXC containers are a favorite option.

etce-lxc eases the task of setting up LXC networks. Given a self-contained LXC Plan File, etce-lxc:

• Generates LXC version appropriate lxc.container.conf style configuration files from templates for multiple containers, optionally situated on multiple physical hosts.
• Generates an init script run by each container on startup. For ETCE, this invariably means starting an sshd instance.
• Creates operating directories for each container in the ETCE Working Directory (under lxcroot).
• Runs lxc-execute to start the containers and creates the Linux Bridges to interconnect them.
• Optionally populates /etc/hosts with Plan File interface addresses and host names.

Demonstration 2 (contained in the 02.hello_lxc subdirectory of the tutorial) runs on this network:

The corresponding LXC Plan File is 02.hello_lxc/doc/lxcplan.xml:

<lxcplan>
  <hosts>
    <host hostname="localhost">
      <bridges>
        <bridge name="br.ctl">
          <ipaddress>
            <ipv4>10.76.0.200/16</ipv4>
          </ipaddress>
        </bridge>
      </bridges>

      <containers>
        <container lxc_name="node-${'%03d' % lxc_index}" lxc_indices="1-2">
          <parameters>
            <parameter name="lxc.tty" value="1"/>
            <parameter name="lxc.pts" value="128"/>
            <parameter name="lxc.console" value="none"/>
            <parameter name="lxc.mount.auto" value="proc sys"/>
          </parameters>
          <interfaces>
            <interface bridge="br.ctl" hosts_entry_ipv4="${lxc_name}">
              <parameter name="lxc.network.type" value="veth"/>
              <parameter name="lxc.network.name" value="backchan0"/>
              <parameter name="lxc.network.flags" value="up"/>
              <parameter name="lxc.network.ipv4" value="10.76.0.${lxc_index}/16"/>
              <parameter name="lxc.network.veth.pair" value="veth.ctl.${lxc_index}"/>
            </interface>
          </interfaces>
          <initscript>#!/bin/bash

# make node for tun device
mkdir /dev/net
mknod /dev/net/tun c 10 200

pidfile="${lxc_directory}/var/run/sshd.pid"
/usr/sbin/sshd -o "PidFile=$pidfile" -o "PermitRootLogin=yes" -o "PasswordAuthentication=no"
          </initscript>
        </container>
      </containers>
    </host>
  </hosts>
</lxcplan>

The hosts element contains one or more host children. Each host corresponds to the host machine where the bridges and containers defined underneath will run - in this case on localhost.

The bridges section names Linux Bridges that will be created on the host, along with their IP address assignment. The bridges section is especially for bridges that require an IP assignment in your test. This will be the case for bridges used to communicate from the host to the containers. In this demo, etce-test makes SSH connections to the containers over br.ctl. Later we'll demonstrate that bridges can also be created implicitly by naming them in the container configuration.

Container configuration is generated from the containers section. Each container element has three children. The first two, parameters and interfaces, lists the parameters used to generate the container configuration files. The items named here are the same as those discussed in the lxc.container.conf man page for general and network container configuration. The third section, initscript, holds the shell script run by the container on launch.

An integral part of the LXC Plan File are the attributes with text fields demarcated by a dollar sign and squiggly braces - 10.76.0.${lxc_index}/16, for example. These fields are mako template strings. The text they enclose is interpreted as Python code and is evaluated by the template engine. The resulting text replaces the marked string. The Python code is evaluated in the context of a set of name/value pairs which changes for each container. In this way, multiple container configurations are generated from the same XML snippet.

The template context is driven by the container element attributes lxc_name and lxc_indices. etce-lxc generates one set of container configuration per index value specified in lxc_indices. For each context, these variables are available for substitution into the template strings:

Two items to note. First, although our current LXC Plan File has one container element, multiple are permitted. etce-lxc requires the lxc_indices values to be unique across all container elements and will reject a Plan File where this rule is violated. Second, the fields listed above are the ones reserved by ETCE for template substitution. Users may also specify substitution fields within the container definition. We'll see examples of this later.

The start_demo.sh script runs ETCE applications in sequence to start the containers, check the SSH connection and run the test. For now, let's run etce-lxc in isolation to start the containers and examine the generated output:

[etceuser@host]$ cd python-etce-tutorial

[etceuser@host]$ sudo etce-lxc start 02.hello_lxc/doc/lxcplan.xml
Bringing up bridge: br.ctl
lxc-execute -f /tmp/etce/lxcroot/node-001/config  -n node-001 -o /tmp/etce/lxcroot/node-001/log -- /tmp/etce/lxcroot/node-001/init.sh 2> /dev/null &
lxc-execute -f /tmp/etce/lxcroot/node-002/config  -n node-002 -o /tmp/etce/lxcroot/node-002/log -- /tmp/etce/lxcroot/node-002/init.sh 2> /dev/null &

The shell capture shows ETCE starting the bridge br.ctl and the two containers node-001 and node-002. etce-lxc works in the lxcroot subdirectory of the ETCE Working Directory. The config and init.sh files passed to each lxc-execute call are the container specific LXC configuration and init script generated from the Plan File. etce-lxc creates a subdirectory in lxcroot for each container.

[etceuser@host]$ ls /tmp/etce/lxcroot
etce.lxc.lock  node-001  node-002

[etceuser@host]$ tree /tmp/etce/lxcroot/node-001
/tmp/etce/lxcroot/node-001
|__ config
|__ init.sh
|__ log
|__ mnt
|__ var
    |__ lib
    |__ log
    |__ run
        |__ sshd.pid

The contents of the config file come directly from the XML values with the templates filled in. Notice, for example, the values for lxc.utsname or lxc.network.ipv4 that are generated in the context of lxc_index=1 and lxc_name=node-001:

[etceuser@host]$ cat /tmp/etce/lxcroot/node-001/config
lxc.uts.name=node-001
lxc.tty.max=1
lxc.pty.max=128
lxc.console.path=none
lxc.mount.auto=proc sys

# br.ctl interface
lxc.net.0.type=veth
lxc.net.0.flags=up
lxc.net.0.ipv4.address=10.76.0.1/16
lxc.net.0.name=backchan0
lxc.net.0.veth.pair=veth.ctl.1
lxc.net.0.link=br.ctl

# loopback interface
lxc.net.1.type=empty

Comparing the Plan File to the resulting configuration, there are a couple important particulars to emphasize:

1. lxc.utsname is not explicitly mentioned in the LXC Plan but it is important enough to be required. Accordingly, ETCE automatically sets this value from the (required) lxc_name container attribute.
2. Similarly, for the first configured network interface, the lxc.network.link value comes from the required interface bridge attribute; the Plan schema makes this a required value also. In our example, br.ctl is one of the bridges listed in the Plan bridges section. However, this is not necessary. A new interface may introduce a new bridge by name. ETCE keeps track of these and instantiates them. Across containers, interfaces that name the same bridge value are connected to the same bridge - by virtue of this mechanism.
3. ETCE added the loopback interface automatically. This is almost always desired so ETCE eliminates the need to configure it manually.

etce-lxc also generates init.sh. It's contents from the LXC Plan initscript section verbatim but with the lxc_directory value filled in for node-001 as /tmp/etce/lxcroot/node-001:

[etceuser@host]$ cat /tmp/etce/lxcroot/node-001/init.sh
#!/bin/bash
# make node for tun device
mkdir /dev/net
mknod /dev/net/tun c 10 200
pidfile="/tmp/etce/lxcroot/node-001/var/run/sshd.pid"
/usr/sbin/sshd -o "PidFile=$pidfile" -o "PermitRootLogin=yes" -o "PasswordAuthentication=no"

Go ahead and run etce-lxc stop to tear down the network (if you examine stop_demo.sh you'll see that this is the only command it runs):

[etceuser@host]$ sudo etce-lxc stop
lxc-stop -n node-001 -k &> /dev/null
lxc-stop -n node-002 -k &> /dev/null
Bringing down bridge: br.ctl

ETCE Wrappers have been mentioned several times. A Wrapper is a dynamically loaded and instantiated Python object that controls the specifics of configuring, starting and stopping a specific application (the wrapped application). Writing a new Wrapper extends ETCE to control a new application. An ETCE Test is essentially a sequenced invocation of Wrappers across multiple hosts.

Writing a new Test requires knowing which Wrappers are installed and how they work. The etce-wrapper application helps with both. Run with the list subcommand to list all installed Wrappers:

[etceuser@host]$ etce-wrapper list
emane.emane
emane.emanecommand
emane.emaneeventd
emane.emaneeventservice
emane.emaneeventtdmaschedule
emane.emanephyinit
emane.emaneshsnapshot
emane.emanetransportd
emane.otapublisher
lte.srsenbemane
lte.srsepcemane
lte.srsmbmsemane
lte.srsueemane
ostatistic.ostatisticsnapshot
otestpoint.otestpointbroker
otestpoint.otestpointd
otestpoint.otestpointrecorder
utils.arpcache
utils.ethtool
utils.gpsd
utils.hello
utils.ifconfig
utils.igmpbridge
utils.ip
utils.iperfclient
utils.iperfserver
utils.mgen
utils.nrlsmf
utils.olsrd
utils.ospfd
utils.ovsctl
utils.ovsdbserver
utils.ovsvswitchd
utils.ripd
utils.sleepwait
utils.smcrouted
utils.sysctlutil
utils.top
utils.zebra

Add the Wrapper name and the verbose flag (-v) to print details about a specific Wrapper:

[etceuser@host]$ etce-wrapper list -v utils.hello
-----------
utils.hello
-----------
path:
  /opt/etce/wrappers
description:

    Demo wrapper that prints "Greeting Recipient!", where 'Greeting'
    is specified by the greeting argument and 'Recipient' is specified
    in the hello.args input file. Set the 'verbose' argument to
    true for a longer greeting.

input file name:
        hello.args
output file name:
        hello.log
arguments:
        greeting
                Greeting to use to address recipient.
                default: Hello
        verbose
                Set to true for a longer greeting.
                default: False                                                 

The printout lists these items:

The meanings of the path and description fields are clear enough. The output file name is typically a log file produced by the application. It is written to the host's subdirectory in the test output directory (by default under /tmp/etce/data as discussed in the previous demo). The input file name and arguments require more explanation.

All Wrappers specify an input file name. During a test step when a Wrapper is prompted to run, it checks for the presence of the input file as a trigger to start the wrapped application. The input file is often the configuration file accepted by the wrapped application. However, this is not necessary. Wrappers require an input file even when the wrapped application doesn't recognize one. In these cases, the input file may contain information consumed by the Wrapper. Or, the file may simply be an empty file to trigger execution (by convention, input filenames that end with .flag signal this case). This behavior is important to understand in writing ETCE Tests as it provides a way to restrict application execution to a subset of nodes by selectively placing input files within the host subdirectories of the Test Directory. We'll return to this point in later demonstrations.

arguments are the configuration items Wrappers expose to control the way the wrapped application runs. They tend to fall into two categories. Some arguments control behavior that varies between tests but is fixed for a given test. Several Wrappers have a daemonize argument that belongs to this category. daemonize controls whether the wrapped application runs as a Linux daemon. Wrappers that run daemons return immediately, otherwise they block until the application completes. Depending on the test scenario, there is only one correct setting for daemonize. In the other group are arguments the user may want to manipulate at runtime on a trial by trial basis. Log level is a common example.

The distinction between the two categories points to a inherent headache in testing. Tests are defined by a large number of input files. One goal in writing tests is to avoid the need to manually edit test files to run common test variations. Manual editing, especially across a large number of tests, leads to mistakes and works against reproducibility. To serve this goal, ETCE Wrapper arguments may be set in two places - in the steps.xml file or in an ETCE Configuration File. Passing values in steps.xml works well for arguments that do not change for a particular test (think daemonize). Arguments that vary from run to run are better specified in a Configuration File; ETCE provides a command line argument for choosing a Configuration File at runtime.

Let's see how this works.

The utils.hello Wrapper's role in life is to print a greeting - "Hello ETCE!" by default - to standard output and to the Wrapper output file. Demonstration 1 runs utils.hello in its default configuration.

Armed now with detailed information about utils.hello, we'll run it again and manipulate its output via its arguments and input file. Here is the Wrapper description again:

description:

    Demo wrapper that prints "Greeting Recipient!", where 'Greeting'
    is specified by the greeting argument and 'Recipient' is specified
    in the hello.args input file. Set the 'verbose' argument to
    true for a longer greeting.

As this demonstration runs on two hosts, node-001 and node-002 (our two LXC containers), we'll configure for customized greetings by specifying different recipients in the input files:

[etceuser@host]$ cat 02.hello_lxc/node-001/hello.args
ETCE node-001

[etceuser@host]$ cat 02.hello_lxc/node-002/hello.args
ETCE node-002

There are also two Wrapper arguments to play with. We'll set both. utils.hello arguments do not fit neatly into the two argument categories discussed above. Nevertheless, we'll demonstrate both vectors for argument passing by setting one in steps.xml and the other in a Configuration File:

[etceuser@host]$ cat 02.hello_lxc/steps.xml
<?xml version="1.0" encoding="UTF-8"?>
<steps>
  <step name="say.hello">
    <run wrapper="utils.hello">
      <arg name="greeting" value="Hi ya"/>
    </run>
  </step>
</steps>

[etceuser@host]$ cat config/config-hello-verbose.xml
<?xml version="1.0" encoding="UTF-8"?>
<testconfiguration>
  <wrapper name="utils.hello">
    <arg name="verbose" value="True"/>
  </wrapper>
</testconfiguration>     

Run the demonstration with start_demo.sh, passing in the Configuration File using the -c option. As for the first demo, this one does not require privileged execution; substitute your user name for etceuser as the SSHUSER parameter.

[etceuser@host]$ ./start_demo.sh -c config/config-hello-verbose.xml etceuser 02.hello_lxc
configfile=config/config-hello-verbose.xml
sshuser=etceuser
demodir=02.hello_lxc
Bringing up bridge: br.ctl
lxc-execute -f /tmp/etce/lxcroot/node-001/config  -n node-001 -o /tmp/etce/lxcroot/node-001/log -- /tmp/etce/lxcroot/node-001/init.sh 2> /dev/null &
lxc-execute -f /tmp/etce/lxcroot/node-002/config  -n node-002 -o /tmp/etce/lxcroot/node-002/log -- /tmp/etce/lxcroot/node-002/init.sh 2> /dev/null &
Waiting for LXCs ...

Checking ssh connections on port 22 ...
host
node-001
node-002

TestCollection:
  02.hello_lxc

Enter passphrase for /home/etceuser/.ssh/id_rsa:
===============
BEGIN "02.hello_lxc" trial 1
Skipping host "localhost". Source and destination are the same.
Trial Start Time: 2019-05-13T20:01:00
----------
testprepper run 2019-05-13T20:01:00 data/etcedemo-02.hello_lxc-20190513T200018/template data/etcedemo-02.hello_lxc-20190513T200018/data
[localhost]
[localhost] Publishing 02.hello_lxc to /tmp/etce/current_test
----------
step: say.hello 2019-05-13T20:01:00 data/etcedemo-02.hello_lxc-20190513T200018/data
[node-002] /bin/echo "Hi ya ETCE node-002! How ya doing?"
[node-001] /bin/echo "Hi ya ETCE node-001! How ya doing?"
trial time: 0000002
----------
Collecting "02.hello_lxc" results.
   Skipping host "localhost". Source and destination are the same.
----------
END "02.hello_lxc" trial 1
===============
Result Directories:
        /tmp/etce/data/etcedemo-02.hello_lxc-20190513T200018

The say.hello step shows the result of our customizations:

step: say.hello 2019-05-13T20:01:00 data/etcedemo-02.hello_lxc-20190513T200018/data
[node-002] /bin/echo "Hi ya ETCE node-002! How ya doing?"
[node-001] /bin/echo "Hi ya ETCE node-001! How ya doing?"

1. "Hello" changed to "Hi ya" - steps.xml
2. "node-001" and "node-002" specific greetings - hello.args
3. An extra "How ya doing?" - config/config-hello-verbose.xml

The output files show the same results:

[etceuser@host]$ cat /tmp/etce/data/etcedemo-02.hello_lxc-20190513T200018/data/node-001/hello.log
Hi ya ETCE node-001! How ya doing?

[etceuser@host]$ cat /tmp/etce/data/etcedemo-02.hello_lxc-20190513T200018/data/node-002/hello.log
Hi ya ETCE node-002! How ya doing?

In the case that a Wrapper argument is set in steps.xml and in the Configuration File, the Configuration File value wins. This is left as a reader exercise.

Stop the demo before going on.

[etceuser@host]$ ./stop_demo.sh