SPICE | Bi Weekly Patching Guide - UMCST/SPICE-user-info GitHub Wiki

Before patching, send a message in the SPICE Discord @CONTRIBUTOR with a time that maintenance will be happening to ensure no one is using a host that will be going down. Send the message at least 5 minutes in advance of starting.

All credentials for SPICE hosts can be found in BitWarden.

Admin VPN setup can be found here

  1. Start the patching journey by opening up...

  2. Make sure to connect through an admin jumpbox external connection while patching

DO NOT patch via the Admin VPN

Connection example: ssh spiceadmin@<garlic/juniper>.e<dcp/mapa>.spicelab.org -p <2222>

NOTE: Connections to emapa and edcp FQDNs are only accepted on University networks(wired, Eduroam), or off campus via the UMS Remote Access VPN(Viscosity client on macOS/Windows, alt config OpenVPN for Linux)

  1. ssh into tarragon from the admin jumpbox

  2. Prepare to patch!

    NOTE: All patching should be done within a tmux session on the jumpbox

    (a) Locate the autopatching playbook(in the home directory of spiceadmin)

    (b) Verify that all hosts in the inventory.ini file are available to be patched at this time

    (c) Run the autopatching playbook

    • The prechecks check each host to ensure there are no active admin VPN sessions, there are no users logged in, there are no active tmux sessions, and that the system doesn't require a reboot. If a reboot is required and patching is selected to continue, the host will reboot before patching. (If there is a recent connection to the Admin VPN, you can confirm with the user that it is okay to patch, by checking the allowed ips section on the error and cross referencing it with the admin VPN tab IPs)

    • Prechecks will run first for all hosts to be autopatched and other hosts to be manually patched. Pay attention to the output to review checks that are failed. Press Enter to acknowledge failed checks for manually patched hosts, and type yes or no for autopatched hosts that failed checks to either continue patching on the host or stop.

    • If a host did not reboot, check to see if it has been rebooted in the last 30 days. If not, reboot the host.

  3. Record the output in the patching form

  4. Manually patch the remaining hosts

    • Patch the jump box being used for bi-weekly patching process last

    • For PCP patching, terraform2 is only accessible via specific addresses, including University networks where the DCP and MAP jump boxes sit

    • Manually patch any hosts that failed autopatching

    • On the Windows hosts perform a choco upgrade and pause Windows updates for the longest extent possible

  5. Record the output in the patching form

  6. BEFORE submitting the form, update the airtable records for each host

    • Under “TASK: linux reboot if needed” , if any have not “skipped” , then they were rebooted, and need the airtable record updated for that

  7. Make sure any essential services, SPICE Admin VPN(WireGuard), Wazuh, etc. are working properly via their user interface

Patching Diagram

⚠️ **GitHub.com Fallback** ⚠️