Bootstrapping M Lab servers - projectbismark/bismark-measurement-server GitHub Wiki

This document explains how to bootstrap a new M-Lab node to start using using Puppet and the bismark-mserver RPM package repos in order to allow automatic and consistent configuration with the remainder of the BISmark M-lab slivers.

Preliminaries

Make sure you have SSH access to the M-Lab nodes. Ensure you've done/can do up to step 4 in this doc: http://www.planet-lab.org/doc/guides/user
Make sure you have SSH access and sudo on the puppetmaster server.
BISmark measurement server slivers' FQDNs are generated by concatenating 'bismark.gt.' to the FQDN of the M-Lab node used in MyPLC, etc. For example, on the M-Lab node mlab1.arn01.measurement-lab.org, the BISmark slice has the name bismark.gt.mlab1.ar01.measurement-lab.org.

Bootstrapping a single new M-Lab machine

Run the following from your workstation:

$ mserver-deployment/bootstrap_single_mlab_node.sh mlab1.arn01.measurement-lab.org --new-cert

Wait for the output from bootstrap_single_mlab_node.sh to resemble the following

info: Creating a new SSL key for bismark.gt.mlab1.arn01.measurement-lab.org
info: Caching certificate for ca
info: Creating a new SSL certificate request for bismark.gt.mlab1.arn01.measurement-lab.org
info: Certificate Request fingerprint (md5): 91:67:72:75:9B:80:66:63:15:32:70:70:13:46:35:56
notice: Did not receive certificate

In another terminal, log into the puppetmaster server and run the following. You should observe similar output:
```
$ sudo puppet cert list
  bismark.gt.mlab1.arn01.measurement-lab.org (91:67:72:75:9B:80:66:63:15:32:70:70:13:46:35:56)
```
You can compare the fingerprints in step 1 and 2 to verify that the CSR requests match, ensuring that puppet is actually signing the certificate you think it's signing.
If the two fingerprints match, go ahead and sign the certificate by running the following:
```
$ sudo puppet cert sign bismark.gt.mlab1.arn01.measurement-lab.org
```
Assuming you don't see any errors in the bootstrap_single_node.sh output (e.g. err: Could not retrieve catalog; skipping run), you're done!

Rebuilding/reinstalling a M-Lab machine that puppet already knows about

The above process will not work if puppet already knows about a machine with the same fully-qualified domain name. This is because it already has a certificate for that node, and the new certificate signing request presented during setup will not match the old certificate. In this case, we need to tell puppet to "forget" about the server first before we proceed. NOTE that this will break any existing puppet configuration on the "forgotten" server.

Log into the puppetmaster server and run the following:
```
$ sudo puppet cert clean HOSTNAME
```
where HOSTNAME is the machine. Again, be very careful that the HOSTNAME is correct. 'clean' will revoke and delete the certificate for that server on the puppetmaster, disabling communication between the puppet client and the puppetmaster.
Proceed to follow the instructions above for "Bootstrapping a single new M-Lab machine".

Bootstrapping many new M-Lab machines at once

If you have a large number of M-Lab machines to bootstrap, you may find the mserver-deployment/mlab_bootstrap_all.sh script handy. It runs the process documented in "Bootstrapping a single new M-Lab machine" above in a loop for a file containing a list of M-Lab node FQDNs.

If you hav a file containing the following,

$ cat arn01_node_names
mlab1.arn01.measurement-lab.org
mlab2.arn01.measurement-lab.org
mlab3.arn01.measurement-lab.org

then you can use mlab_boostrap_all.sh as follows, following steps 2-3 from "Bootstrapping a single new M-Lab machine" for each node as necessary:

$ ./mlab_bootstrap_all.sh arn01_node_names

Testing a series of freshly bootstrapped machines

Ideally, you should try all of the services running on the newly-bootstrapped machines. As a very basic test to check that the UDP iperf server is running, and that the nodes have reasonable bandwidth available (~100Mbps), you can run the following smoke test:

$ for n in $(cat arn01_node_names); do iperf -c bismark.gt.$n -u -b 100M; done;