Testbed deployment - maggiemsft/SONiC GitHub Wiki

Testbed servers

In testbed preparation and planning step, we already have Ubuntu server and network configured correctly.

Now in test server:

Add Docker's official GPG key

   $ curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo apt-key add -

Setup docker registry for PTF and sonic-mgmt dockers

PTF docker is used in virtual testbed topology connecting to SONiC DUT; and send and receive packets to test DUT data plane.

sonic-mgmt docker is a testbed ansible running environment docker for user to deploy testbed and run test cases.

Ansible playbook in sonic-mgmt repo requires to setup ansible and various library dependencies.

We build a sonic-mgmt docker that packs all dependencies and tools within docker. When you build and run the docker, you have a running environment that is exactly the same with all different community numbers with the same test environment. You run ansible playbook inside the docker so we could have more consistent results.

How to setup docker registry

Build PTF docker

git clone --recursive https://github.com/Azure/sonic-buildimage.git
make configure PLATFORM=generic
make target/docker-ptf.gz

Then upload docker-ptf to the docker registry.

Build sonic-mgmt docker

git clone --recursive https://github.com/Azure/sonic-buildimage.git
make configure PLATFORM=generic
make target/docker-sonic-mgmt.gz

Then upload sonic-mgmt to the docker registry.

Pre-built sonic-mgmt can also be downloaded from here.

Run sonic-mgmt docker

docker load -i target/docker-sonic-mgmt.gz
docker run -it docker-sonic-mgmt bash
cd ~/sonic-mgmt

From now on, all steps are running inside the sonic-mgmt docker.

Latest sonic-mgmt repo is cloned into sonic-mgmt docker under '/var/[your-login-username]/sonic-mgmt`. Once you are in the docker, you need to modify the testbed configuration files to reflect your lab setup.

To avoid lose your sonic-mamagement repo when docker was destroied, you may also clone the sonic-mgmt repo into your test server host directory. Then mount your host directory into sonic-mgmt docker. docker volumes

  • Based on what we have been prepared in last section of all inventory files and testbed information, within your local sonic-mgmt repo, update all inventory files for VMs, SONiC DUTs, fanout switches in ansible inventory folder per your lab setup.

    • Server

      • Update server inventory file in ansible/tetbed_inv/servers.
      • Check that ansible could reach this device by command ansible -m ping -i veos vm_host_1.

    • VM

      • Download vEOS image from arista.
      • Copy below image files to ~/veos-vm/images on your testbed server. Aboot-veos-serial-8.0.0.iso vEOS-lab-4.15.9M.vmdk
      • Update VM IP addresses ansible/tetbed_inv/vms inventory file. These IP addresses should be in the management subnet defined above.

    • Fanout switches

      Update fanout ansible/testbed_inv/fanouts inventory file

    • SONiC DUTs

    Update SONiC DUT ansible/testbed_inv/sonics inventory file

  • Update Secret file secrets.yml[ansible/group_vars/all/secrets.yml] to have all your testbed related credentials.

Bring up all VMs for individual server

./testbed-cli.sh start-vms server_1 password.txt

please note: Here password.txt is the ansible vault password file name/path.

Ansible allows user use ansible vault to encrypt password files. By default, this shell script require a password file. If you are not using ansible vault for, just create an empty file and pass the filename to the command line. touch password.txt The file name and location is created and maintained by user.

Check that all VMs are up and running: ansible -m ping -i veos server_1

Define Testbed topology

ansible/testbed_inv/testbed_topology.yml

This file defines how DUT connect to virtual topology.

topology_1:
  name: meaningful_name_for_user
  internal_name: tp_1 (unique name used in create backplane bridge name, <= 8 character)
  dut: sonic_dut_1
  topo: t0

create/generate actual testbed assignment file

Based on above testbed_topology, pick available vlans and vms, use script or playbook generate testbed_deployment file, saved in server

topology_1:
  name: meaningful_name_for_user
  internal_name: tp_1 (unique name used in create backplane bridge name, <= 8 character)
  dut: sonic_dut_1
  topo: t0
  starting_vlan: 2000
  starting_vm: VM0100
  server: server_1

Create/generate testbed actual connections files

Based on sonic_dut_in_topology, vlan assignment, we could be able to create actual connections file:

ansible/files/testbed_connection.yml

question need consider: how to count interface_name on fanout side(hardwaresku for fanout is random depends on connection DUT port break out)? How to dynamically deploy fanout switch? when connect testbed, need run time move and add vlan on interfaces, need "config load -j" ? traffic interrupt?
How to trigger? need automatically trigger? phase-1: manually run update file like create_graph before? Check in testbed_connection.yaml to server.

fanout:
    leaffanout:
  	leaf_fanout_1:
  	  Ethernet0: 
  		  peer_device: sonic_dut_1
  		  peer_port: Etherent0
  		  port_phy_speed: 
  		  vlan_mode:
  		  vlan_id: 
  	  Ethernet4:
  		  ... 
  	  Ethernet124: 
  		  peer_device: root_fanout
  		  peer_port: eth7
  		  port_phy_speed: 
  		  vlan_mode:
  		  vlan_id: 

  	leaf_fanout_2:
  	  Ethernet0: ...

    rootfanout:
      root_fanout:
  		Ethernet0:
  		  peer_device: server_1
  		  peer_port: p4p1
  		  port_phy_spped: 40
  		  vlan_mode: trunk
  		  vlan_id: 
  		Etherenet4:
  		  peer_device: fanout_leaf_1
  		  peer_port: Eth64/1
  		  peer_phy_speed: 100
  		  vlan_mode: trunk
  		  vlan_id: 100-132,247-256

Deploy fanout switch and Vlan

Based on sonic_dut_in_topology, vlan assignment, we could be able to create actual connections file: ansible/files/testbed_connection.yml

  1. initial deploy all fanout switches based on the connection file.
  2. when adding new dut or change connections, on demand deploy fanout switches need change
  3. What's the diff trigger? ?

Deploy topology

Based on ansible/testbed_inv/testbed_topology.yml file, we could generate testbed assignment file which described in preparation section. Then we should be able to connect the virtual topology to SONiC DUT.