Automating Staged Upgrades - robedi/PowerShell GitHub Wiki

Automating Staged Upgrades for Octopus Deploy

This guide provides instructions on how to use the PowerShell script to automate a staged upgrade process for an Octopus Deploy server. The script handles downloading, backing up, upgrading, and restarting the server incrementally through several versions, ensuring a smooth and stable upgrade with minimal downtime.

Original script location: Upgrading single node Octopus Deploy instances

Prerequisites

Before running the script, make sure you have the following:

  • Octopus Deploy API Key: This is required for authentication and managing the server.
  • Administrative Access: Ensure the script is run with admin privileges on the Octopus Deploy server.
  • Backup Permissions: Permissions to write backups to the specified directories.
  • PowerShell 5.1 or Later: PowerShell is required to run the script.

Static Badge Static Badge

Steps to Configure and Run the Script

1. Clone the Repository or Download the Script

  • Clone this repository or download the script file to your Octopus Deploy server.
  • Make sure the script is accessible and ready to be executed with appropriate permissions.

2. Update the Script with Your Configuration

  • Open the script in your preferred text editor (e.g., Visual Studio Code) and update the following placeholders to match your environment:
Variable Name Description Example
$url URL of your Octopus Deploy Web Portal https://octopus.yourdomain.com
$apiKey Your Octopus API key for authentication API-YOUR-KEY
$octopusDeployDatabaseName The name of your Octopus Deploy database OctopusDeploy
$sqlBackupFolderLocation Directory or network share where database backups will be saved \\Server\Backups\Database
$fileBackupLocation Directory or network share where file backups (logs, artifacts, etc.) will be saved \\Server\Backups\Files
$downloadDirectory Temporary folder or network share where installer MSI files will be downloaded ${env:Temp}
$settleTimeInMinutes Time (in minutes) to wait after each upgrade to allow the server to stabilize 30

3. Build the Staged Versions List

  • Visit the Octopus Server Release History and identify your current installed version.
  • For a stable, staged upgrade, follow these steps:
    • Perform minor version upgrades (e.g., 2019.11.x to 2019.12.x).
    • Focus on upgrading to the latest patch for each minor version (e.g., 2019.12.14).
  • Add each final version for every minor upgrade to the $stagedVersions array, starting from the oldest and working towards the newest:
$stagedVersions = @(
   "Octopus.2023.2.13580-x64.msi",
   "Octopus.2023.3.13361-x64.msi",
   "Octopus.2023.4.8624-x64.msi",
   "Octopus.2024.1.13034-x64.msi",
   "Octopus.2024.3.12741-x64.msi"
)
  • Save the script after updating the staged versions list.

4. Run the Script

  • Open PowerShell ISE with administrative privileges on the Octopus Deploy server.
  • Navigate to the location where the script is saved and execute it:
.\AutomatingStagedUpgrades.ps1

[!TIP] Monitor the process and stop the script to either, check if the instance is functioning as expected or troubleshoot an issue.

The script will:

  1. Check for Updates: It compares the current version of Octopus with the latest available version.
  2. Backups: Before each major version upgrade, the script backs up critical files (artifacts, logs, packages, telemetry) and the database.
  3. Upgrade Execution: The script downloads and applies the necessary upgrades incrementally.
  4. Stabilization Period: After each upgrade, the script waits for the server to stabilize before moving on to the next version.

Key Sections in the Script

1. Variables and Configurations

Ensure the following variables are correctly set in the script:

  • $url: Your Octopus Deploy instance URL.
  • $apiKey: Your API key for authentication.
  • $octopusDeployDatabaseName: The name of your Octopus Deploy database.
  • $sqlBackupFolderLocation: The location where the database backup will be saved.
  • $fileBackupLocation: The location where file backups will be saved (e.g., logs, artifacts).
  • $downloadDirectory: The folder where installer files will be downloaded.
  • $settleTimeInMinutes: The wait time between version upgrades (default: 30 minutes).

2. Version Detection and Comparison

The script fetches the current version and compares it to the latest available version using the following logic:

$currentVersion = (Invoke-RestMethod "$Url/api").Version
$versions = Invoke-RestMethod "https://octopus.com/download/upgrade/v3"
$upgradeVersion = $versions[-1].Version

It also uses the Compare-Version function to determine if the current version is older and needs to be upgraded.

3. Backup Process

The script checks if a major upgrade is detected, and if so, backs up critical files using robocopy:

Start-Process -FilePath "robocopy" -ArgumentList "$($serverFolders.ArtifactsDirectory) $fileBackupLocation\Artifacts /mir"

It also creates a backup of the Octopus Deploy database:

$command.CommandText = "BACKUP DATABASE [$octopusDeployDatabaseName] TO DISK = '$backupFileFullPath' WITH FORMAT;"

4. Maintenance Mode

Before performing an upgrade, the script places the server into maintenance mode:

Invoke-RestMethod -Uri "$url/api/maintenanceconfiguration" -Headers @{'X-Octopus-ApiKey' = $apiKey} -Body (@{ Id = "maintenance"; IsInMaintenanceMode = $true } | ConvertTo-Json)

After the upgrade, the server is taken out of maintenance mode:

Set-OctopusOutOfMaintenanceMode -url $url -apiKey $apiKey

5. Upgrade Process

The script downloads and installs each version in the $stagedVersions array incrementally:

$msiExitCode = (Start-Process -FilePath "msiexec.exe" -ArgumentList "/i $msiToInstall /quiet" -Wait -PassThru).ExitCode

6. Post-Upgrade Wait Time

The script waits for a specified period ($settleTimeInMinutes) after each upgrade to allow the server to stabilize:

Start-Sleep -Seconds ($settleTimeInMinutes * 60)

Error Handling

The script includes error handling and retry logic:

  • If a network error occurs while connecting to the server, it retries the connection several times before exiting.
  • If robocopy or database backup commands fail, the script exits and provides a clear error message for debugging.

Testing and Best Practices

  • Test on a Non-Production Instance: Always run the script on a test instance or staging environment before applying it to a production server.
  • Manual Verification: After running the script, manually verify the status of the Octopus Deploy server to ensure all upgrades were successful.
  • Regular Backups: In addition to the backups performed by the script, ensure you have regular database and file backups as part of your disaster recovery plan.

By following this work instruction, you will be able to automate the staged upgrade process of an Octopus Deploy server, ensuring that each upgrade step is applied incrementally with minimal downtime and risk.