We do not use the same protocol to update the license or software syslog server that Systems uses to update clusters.

They rebuild all nodes from scratch.  We just update.  Below is the procedure to follow for each of flux-license1.miserver.it.umich.edu and arc-soft-vm.miserver.it.umich.edu.

1. At least one week before the planned update, make sure you can log into the VMware vSphere Client.  It is at https://vcenter.umich.edu/ui and it uses Duo.  Enter your uniqname in the user box, then in the password box, enter your kerberos password, a comma, then your YubiKey code.  If you get Duo Pushes as your default, then it may push you if you leave off the comma and Duo code.  Try it.

2. Before applying an update of any kind, log into the vSphere client and make a snapshot of the virtual machine.  From the main menu/dashboard, you want VMs and Templates from the main dashboard.  If there isn't one of those, then click on the VM logo in the upper-left corner to try to get to where there is one.  That might take you to a screen with some buildings on it.  You want the VMs tab, not the Summary tab that shows the buildings.  From there, you can click on whichever you are working on.
   
   To make a snapshot, use the Actions menu, Snapshots, Manage Snapshots.  From there you can see whether there are old snapshots left.  You want to clear those out first.  That's scary.  See below for some notes on clearing snapshots.  Once you have cleared those out, go back to Actions, Snapshots, and Take Snapshot and take it.

3. Once the snapshot has been taken, you need to have an updated Ansible playbook to work with.  Systems will have updated most things, but you will need to update (or get updated) flux-admin09:/etc/ansible/license_server/group_vars/all/all through the usual Git clone/hotfix/PR flow.  That will contain the latest CentOS version and the latest ISO version.  Contents should look like this, with suitable versions.
   \

   centosVersion: 7.7.2001
centosISOversion: 7.6.1810
   \

4. Once that change has been done, you should run this (change the hostname for the other machine, of course)
   
   # ansible-playbook -i license_server -l arc-soft-vm.miserver.it.umich.edu utilityPlaybook/yum_repo.yml -D -C
   
   from /etc/ansible on flux-admin09 as root.  The -C says check it, but don't run, the -D is for difference (not dry-run or debug) and sort of shows you what's changing.  You then need to run it a second time with just -D.

5. From a separate terminal, log into the the VM and run yum check-update to make sure it looks OK.  What's OK?  Expect quite a few updates, especially look for updates to the kernel.  That should be there.  If not, check the group_vars file mentioned above and make sure that it's correct.

6. Once that has been done, you then need to run
   
   # ansible-playbook -i license_server -l arc-soft-vm.miserver.it.umich.edu utilityPlaybook/runYumUpdate.yml -D
   
   No check or dry-run for this puppy; you Just Do It.  This can take a while.  It will display a lot of messages about FAILING -- don't be fooled, it's working.  It got to 24 left of 60 retries to finish last time I ran it.  Don't ask me why it can't spell 'updater'.
   \

   PLAY RECAP **************************************************************************************************
arc-soft-vm.miserver.it.umich.edu : ok=2 changed=2 unreachable=0 failed=0

   Monday 09 March 2020 19:02:50 -0400 (0:18:36.583) 0:18:39.110 **********
===============================================================================
YUM - check on updatre task ----------------------------------------------------------------------- 1116.58s
Update all yum installed packgages ------------------------------------------------------------------- 2.24s

7. On the VM being updated, run yum check-update again, and there shouldn't be a list.  Reboot it.

8. Wait, and log back in to make sure it worked and the VM is actually working.