ArcGIS Enterprise on Kubernetes in AKS - Esri/arcgis-gitops GitHub Wiki

This walkthrough will guide you through the process of initial deployment and disaster recovery of ArcGIS Enterprise on Kubernetes 11.5 deployment in Microsoft Azure public cloud using GitHub Actions.

The walkthrough uses azure/arcgis-site-core and azure/arcgis-enterprise-k8s templates.

Duration: about 2 hours

Before you begin, you need to have the following resources and accounts:

• GitHub.com user account
• Service principal in Microsoft Azure account with Owner role
• Microsoft Azure user account to administer Azure Managed Grafana
• Docker Hub account that has access to the private repositories with ArcGIS Enterprise on Kubernetes container images
• Authorization file for ArcGIS Enterprise on Kubernetes 11.5
• Helm charts for ArcGIS Enterprise on Kubernetes 11.5 (ArcGIS_Enterprise_on_Kubernetes_Helm_Charts_1.5.0_195642.tgz)
• Domain name for the ArcGIS Enterprise site
• SSL certificate files (certificate, private key, and chain files in PEM format) for the ArcGIS Enterprise site domain name

The authorization file and Helm charts for ArcGIS Enterprise on Kubernetes can be downloaded from My Esri.

This step creates new private GitHub repository for ArcGIS Enterprise site from the template repository.

Open https://github.com/esri/arcgis-gitops URL in a web browser, click on Use this template green button, and select "Create a new repository" from the dropdown menu.

Select the repository owner, enter "arcgis-aks" as repository name and "ArcGIS Enterprise on Kubernetes in AKS" as description, change the visibility to "Private", and click "Create repository" button.

This step installs Azure CLI in a GitHub codespace workspace and uses it to create an Azure storage account and a blob container for Terraform backend that will be used by the workflows.

In the new repository, click the green "<> Code" button, switch the tab to "Codespaces", and click the Create codespace on main button. The codespace will be created and opened in a new browser tab.

In the codespace, click on the "Terminal" tab in the bottom panel and run the following commands:

curl -sL https://aka.ms/InstallAzureCLIDeb | sudo bash

Run the following commands, replacing <client id>, <client secret>, and <tenant id> by the service principal's credentials:

az login --service-principal --username <client id> --password <client secret> --tenant <tenant id>

Get all subscriptions for the tenant:

az account subscription list

Output:

[
  {
    "authorizationSource": "RoleBased",
    "displayName": "Azure subscription",
    "id": "/subscriptions/XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXXXXX",
    "state": "Enabled",
    "subscriptionId": "XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXXXXX",
    "subscriptionPolicies": {
      "locationPlacementId": "Public_2014-09-01",
      "quotaId": "MSDNDevTest_2014-09-01",
      "spendingLimit": "Off"
    }
  }
]

Note "subscriptionId" value of the subscription you want to use.

Check that the service principal is in the Owner role in the subscription:

az role assignment list  --all --assignee <client id> --query "[].{roleDefinitionName:roleDefinitionName scope:scope}" --output tsv

Output:

Owner   /subscriptions/XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXXXXX

Set the current subscription:

az account set --subscription <subscription id>

Create a resource group for the Azure storage account in East US region by running the following command:

az group create --name arcgis-gitops --location eastus

Names and display names of the available Azure locations can be listed by running az account list-locations --query "[].{name:name displayName:displayName}" --output tsv command.

Run the following command replacing <storage account name> with a globally unique name:

az storage account create --name <storage account name> --resource-group arcgis-gitops

NOTE: Storage account name must be between 3 and 24 characters in length and use numbers and lower-case letters only.

Replace <storage account name> with the storage account name and run the following command to create blob container "tfbackend" in the storage account:

az storage container create --name tfbackend --account-name <storage account name> --auth-mode login

Replace <client id>, <subscription id>, and <storage account name> with the service principal client ID, subscription ID, and the storage account name in the following command and run the command to grant the service principal required access to the blob container:

az role assignment create --role "Storage Blob Data Owner" --assignee <client id> --scope /subscriptions/<subscription id>/resourceGroups/arcgis-gitops/providers/Microsoft.Storage/storageAccounts/<storage account name>/blobServices/default/containers/tfbackend

This step configures and validates secrets and variables in the repository settings used by the GitHub Actions workflows to retrieve Azure and Docker Hub credentials and other properties.

Open the new repository settings by clicking on the Settings tab in the repository, then click on "Secrets and Variables" menu item from "Security" section of the left sidebar, and select "Actions" from the dropdown menu.

• Click on "New repository secret" button, enter AZURE_CLIENT_ID into the "Name" field, enter the service principal client ID into the "Secret" field, and click "Add secret" button.
• Add AZURE_CLIENT_SECRET secret with the service principal client secret.
• Add AZURE_TENANT_ID secret with the Microsoft Entra tenant ID.
• Add CONTAINER_REGISTRY_USER secret with the Docker Hub username.
• Add CONTAINER_REGISTRY_PASSWORD secret with the Docker Hub user password.

Switch the "Secrets"/"Variables" tab to "Variables".

• Click on "New repository variable" button, enter AZURE_DEFAULT_REGION into the "Name" field, enter "East US" into the "Value" field, and click "Add variable" button.
• Add AZURE_SUBSCRIPTION_ID variable with the Azure subscription ID.
• Add TERRAFORM_BACKEND_STORAGE_ACCOUNT_NAME variable with the Azure storage account name of Terraform backend.
• Add TERRAFORM_BACKEND_CONTAINER_NAME variable set to "tfbackend".

If you need to use an Azure region other than East US, check if the region is supported by Application Gateway for Containers.

Click on "Actions" tab in the repository, select "validate-settings-azure" workflow on the left sidebar, click "Run workflow" on the right, and click "Run workflow" button to run the workflow. Reload the web page and wait until the workflow run is completed. The workflow will succeed if the credentials are valid.

This step copies the required workflows to .github/workflows directory and lists the available workflows in the repository.

Switch back to the codespace browser tab and run the following commands:

cp -r azure/arcgis-site-core/workflows/* .github/workflows/

cp -r azure/arcgis-enterprise-k8s/workflows/* .github/workflows/

git add --all

git commit -m "Enable workflows"

git push origin main

List the available workflows in the repository:

gh workflow list

Output:

NAME                               STATE   ID       
enterprise-k8s-azure-backup        active  138163280
enterprise-k8s-azure-destroy       active  138163281
enterprise-k8s-azure-image         active  138163282
enterprise-k8s-azure-ingress       active  138163283
enterprise-k8s-azure-organization  active  138163284
enterprise-k8s-azure-restore       active  138163285
enterprise-k8s-azure-test          active  138163286
site-core-azure-destroy            active  138163287
site-core-azure                    active  138163288
site-k8s-cluster-azure-destroy     active  138163289
site-k8s-cluster-azure             active  138163290
validate-settings-aws              active  137938401
validate-settings-azure            active  137938402

This step updates the configuration files used by the workflows.

In the EXPLORER sidebar of the codespace window, click on "config" folder, then right-click on "certificates" folder, and select "Upload..." from the context menu. In the file selector dialog window navigate to the SSL certificate files location, select the certificate, private key, and chain files, and click "Open" button to upload the files to the workspace.

Click again on "config" folder, then on "azure" subfolder, and then on "arcgis-enterprise-k8s" subfolder. Double-click on "ingress.tfvars.json" file to open it in the editor.

Change "deployment_fqdn" value to the fully qualified domain name of the ArcGIS Enterprise site.

Replace "tls_certificate_path", "tls_private_key_path", and "ca_certificate_path" properties with the uploaded SSL certificate file paths.

In the EXPLORER sidebar of the codespace window, navigate to "azure/arcgis-enterprise-k8s/organization/helm-charts/arcgis-enterprise" folder and select "Upload..." from the context menu. In the file selector dialog window navigate to the Helm charts location, select ArcGIS_Enterprise_on_Kubernetes_Helm_Charts_1.5.0_195642.tgz file and click "Open" button to upload the file to the workspace.

Change the current folder to the Helm charts subfolder:

cd azure/arcgis-enterprise-k8s/organization/helm-charts/arcgis-enterprise/

Extract the Helm charts from the archive:

tar -xzf ArcGIS_Enterprise_on_Kubernetes_Helm_Charts_1.5.0_195642.tgz

Rename the extracted subfolder to the Helm chart version:

mv arcgis-enterprise 1.5.0

Delete the Helm charts archive:

rm ArcGIS_Enterprise_on_Kubernetes_Helm_Charts_1.5.0_195642.tgz

Go back to the workspace root folder:

cd /workspaces/arcgis-aks

In the EXPLORER sidebar of the codespace window, click on "config" folder, then right-click on "authorization" folder, select "New Folder..." from the context menu, and enter "11.5" as the folder name. Right-click on the new "11.5" folder, and select "Upload..." from the context menu. In the file selector dialog window navigate to the ArcGIS Enterprise on Kubernetes authorization file location, select the file and click "Open" button to upload the file to the workspace.

Open config/azure/arcgis-enterprise-k8s/organization.tfvars.json file in the editor and update the following properties:

• Replace "authorization_file_path" property value with the uploaded authorization file paths "~/config/authorization/11.5/<authorization file name>".

• Set "deployment_fqdn" property to the ArcGIS Enterprise site fully qualified domain name.

• Set "admin_first_name", "admin_last_name", "admin_username", "admin_password", "admin_email", "security_question_index", and "security_question_answer" to the ArcGIS Enterprise primary administrator account properties.

The primary administrator user name must be at least six characters in length. The only special characters allowed are the at sign (@), dash (-), dot (.), and underscore (_). The primary administrator password must be at least eight characters in length. It must contain at least one alphabet letter (uppercase or lowercase), at least one digit, and at least one special character. All special characters are allowed.

Change the "passcode" property value in backup.vars.json and restore.vars.json config files to a passcode that will be used when restoring the backup.

Replace the "deployments" property value in config/azure/site-index.json file with ["arcgis-enterprise-k8s"].

git add --all

git commit -m "Update config files"

git push origin main

Click on "Actions" tab in the repository, select "verify-site-config-azure" workflow on the left sidebar, click "Run workflow" on the right, and click "Run workflow" button to run the workflow. Reload the web page and wait until the workflow run is completed. The workflow will succeed if no errors are found in the config files.

This step runs the workflows to provision the core Microsoft Azure infrastructure and create an Azure Kubernetes Service (AKS) cluster.

If a workflow run fails, correct the error and run the workflow again.

Click on "Actions" tab in the repository, refresh the web page, select "site-core-azure" workflow on the left sidebar, click "Run workflow" on the right, and click "Run workflow" button.

Once the workflow run is complete, run the following command in the codespace terminal to list the created Azure resources:

az resource list --resource-group arcgis-infrastructure-core --query "[].{type:type name:name}" --output tsv

Output:

Microsoft.Network/publicIPAddresses     arcgis-nat
Microsoft.Network/networkSecurityGroups arcgis-internal
Microsoft.KeyVault/vaults       arcgis
Microsoft.Storage/storageAccounts       siteced309d2da222641
Microsoft.Network/networkSecurityGroups arcgis-bastion
Microsoft.Network/virtualNetworks       arcgis
Microsoft.Network/natGateways   arcgis
Microsoft.Network/networkSecurityGroups arcgis-private
Microsoft.Network/privateDnsZones       privatelink.blob.core.windows.net
Microsoft.Network/publicIPAddresses     arcgis-bastion
Microsoft.Network/privateDnsZones/virtualNetworkLinks   privatelink.blob.core.windows.net/blob-private-dns-zone-vnet-link
Microsoft.Network/bastionHosts  arcgis-bastion
Microsoft.Network/privateEndpoints      siteced309d2da222641-private-endpoint
Microsoft.Network/networkInterfaces     siteced309d2da222641-private-endpoint.nic.418e4e78-1bdf-48e9-a892-15bf61f46e1d

Run "site-k8s-cluster-azure" workflow to create an AKS cluster that meets ArcGIS Enterprise on Kubernetes system requirements.

Once the workflow run is complete, run the following command in the codespace terminal to list the created Azure resources:

az resource list --resource-group arcgis-k8s-cluster --query "[].{type:type name:name}" --output tsv

Output:

Microsoft.ContainerService/managedClusters      arcgis
Microsoft.ManagedIdentity/userAssignedIdentities        azure-alb-identity
Microsoft.Monitor/accounts      arcgis-enterprise
Microsoft.ServiceNetworking/trafficControllers  arcgis
Microsoft.Network/privateDnsZones       privatelink.azurecr.io
Microsoft.ContainerRegistry/registries  arcgis214c33f19674fa7f
Microsoft.AlertsManagement/prometheusRuleGroups UXRecordingRulesRuleGroup - arcgis-enterprise
Microsoft.Dashboard/grafana     arcgis
Microsoft.AlertsManagement/prometheusRuleGroups NodeRecordingRulesRuleGroup - arcgis
Microsoft.AlertsManagement/prometheusRuleGroups KubernetesRecordingRulesRuleGroup - arcgis
Microsoft.Network/privateEndpoints      arcgis214c33f19674fa7f-private-endpoint
Microsoft.Network/privateDnsZones/virtualNetworkLinks   privatelink.azurecr.io/acr-private-dns-zone-vnet-link
Microsoft.Network/networkInterfaces     arcgis214c33f19674fa7f-private-en.nic.43d62ffc-619a-4dff-85af-0806094bd618
Microsoft.ServiceNetworking/trafficControllers/associations     arcgis/arcgis

Retrieve the AKS cluster access credentials into the .kube/config file so kubectl can use them:

az aks get-credentials --resource-group arcgis-k8s-cluster --name arcgis --overwrite-existing

Test connection to the AKS cluster:

kubectl get nodes

Output:

NAME                              STATUS   ROLES    AGE   VERSION
aks-default-21074325-vmss000000   Ready    <none>   13m   v1.30.7
aks-default-21074325-vmss000001   Ready    <none>   13m   v1.30.7
aks-default-21074325-vmss000002   Ready    <none>   13m   v1.30.7
aks-default-21074325-vmss000003   Ready    <none>   13m   v1.30.7

Replace <user object id with the Azure user account object ID and <subscription id> with the Azure subscription ID and run the following command to assign the user "Grafana Admin" role in "arcgis" Azure Managed Grafana instance:

az role assignment create --assignee-object-id "<user object id>" --assignee-principal-type user --role "Grafana Admin" --scope "/subscriptions/<subscription id>/resourceGroups/arcgis-k8s-cluster/providers/Microsoft.Dashboard/grafana/arcgis"

If the service principal has permissions to query Microsoft Entra ID, you also can use the user's sign-in name instead of client ID with --assignee parameter.

Retrieve the Grafana endpoint URL from the workflow run log or from the output of the following command:

az grafana show --resource-group arcgis-k8s-cluster --name arcgis

Open the URL in a web browser and login to Grafana with the Azure Entra ID user.

This step runs the workflows to build Enterprise Admin CLI container image, create Kubernetes namespace for the deployment and configure ingress, create ArcGIS Enterprise organization, test and backup the deployment.

Run "enterprise-k8s-azure-image" workflow that builds Enterprise Admin CLI container image and pushes it to the private ACR repository of the AKS cluster.

Run "enterprise-k8s-azure-ingress" workflow that creates the ingress resources for the ArcGIS Enterprise on Kubernetes deployment.

After the workflow run is complete, retrieve DNS name of the load balancer created by the workflow from the workflow run log and create a CNAME record for it within the DNS server of the ArcGIS Enterprise domain name.

Run "enterprise-k8s-azure-organization" workflow to deploy ArcGIS Enterprise on Kubernetes in the AKS cluster and create an ArcGIS Enterprise organization.

After the workflows run is complete, the ArcGIS Enterprise site will be accessible at the deployment FQDN https://<deployment FQDN>/arcgis/manager.

The first run of the workflow may fail with BackoffLimitExceeded error. Run the workflow again if that happened.

Run "enterprise-k8s-azure-test" workflow.

Run "enterprise-k8s-azure-backup" workflow.

This step destroys the Azure resources created by the walkthrough.

Run "enterprise-k8s-azure-destroy" workflow with "Delete namespace" option checked.

Run "site-k8s-cluster-azure-destroy" workflow.

Run "site-core-azure-destroy" workflow.

Run the following command in the codespace terminal to delete "arcgis-gitops" resource group:

az group delete --name arcgis-gitops

The walkthrough demonstrates only the basic capabilities of ArcGIS Automation using GitHub Actions. Check instructions of the templates and workflows for more advanced features and options.