SPICE | Bi Weekly Patching Guide - UMCST/SPICE-user-info GitHub Wiki

Before patching, send a message in the SPICE Discord @CONTRIBUTOR with a time that maintenance will be happening to ensure no one is using a host that will be going down. Send the message at least 5 minutes in advance of starting.

All credentials for SPICE hosts can be found in BitWarden.

Admin VPN setup can be found here

  1. Start the patching journey by opening up...

  2. Make sure to connect through an admin jumpbox external connection while patching

DO NOT patch via the Admin VPN

Connection example: ssh [email protected]<garlic/juniper>.e<dcp/mapa>.spicelab.org -p <2222>

NOTE: Connections to emapa and edcp addresses are only allowed on campus, or off campus via the UMS Remote Access VPN(Viscosity client on macOS/Windows)

  1. ssh into tarragon from the admin jumpbox

  2. Prepare to patch!

    NOTE: All patching should be done within a tmux session on the jumpbox

    (a) Locate the autopatching playbook(in the home directory of spiceadmin)

    (b) Verify that all hosts in the inventory.ini file are available to be patched at this time

    (c) Run the autopatching prechecks playbook to verify the following:

    • There are no active Admin VPN sessions

    • There are no users logged in

    • There are no active tmux sessions

    • The system doesn't require a reboot

      The playbook will present this information, if found, near the end of its execution.

    (d) Run the autopatching playbook

    • If a host did not reboot, check to see if it has been rebooted in the last 30 days. If not, reboot the host.

  3. Record the output in the patching form

  4. Manually patch the remaining hosts

    • Patch the jumpbox last

    • Patch PCP hosts via terraform2 host

    • Manually patch any hosts that failed autopatching

  5. Record the output in the patching form

  6. Before sending off the form, update the airtable records for each host

    • Under “TASK: linux reboot if needed” , if any have not “skipped” , then they were rebooted, and need the airtable record updated for that

  7. Make sure any essential services, Wireguard, Wazuh, etc. are working properly via their user interface

⚠️ **GitHub.com Fallback** ⚠️