Azure Automation - llewellyn-marriott/notes GitHub Wiki

Azure Automation

Here I have documented my experiences with Azure Automation. Please let me know if I have gotten anything here wrong. To start with a few points:

  • I am a lazy person, I love automation and writing PowerShell scripts to do things for me.
  • When I first found out about Azure Automation (AA) I thought it sounded great.
  • After trying to use AA in production I have been left disappointed.
  • Here are the issues I ran into while using and also what I did to workaround these problems.

What is?

Azure Automation (AA)

The service from Microsoft (MS) that can run scripts on a schedule for you. Code can be:

  • PowerShell
  • Python

Azure Automation Account (AAA)

The Azure resource that holds Runbooks and associated resources

Runbook

Code that can be ran with AA Can be executed by:

  • Schedule
  • Webhook
  • Manually
  • Watcher Tasks (Sort of, more about that below)

The problems I had

Now on to the issues I experienced, and the things that disappointed me.

Private endpoints

  • If you want to connect to AA without going out to the internet, you need to set up a Private Endpoint connection. This is fine.
  • You can only use Private Endpoints if you use a Hybrid Runbook Worker. I really wish they could be used with the Azure sandbox also.
    • If AA could start/stop containers in your own environment to execute Runbooks that would be amazing too.

Schedule granularity

  • The shortest interval you can use for runbooks is hourly.
  • You can get around this by creating multiple schedules to the same Runbook and offset them by a smaller amount.
  • I feel like it is trivial to allow for more granularity here.

Disaster recovery

  • If you want to make sure your AA tasks still run if a region goes down, the official MS method is to have a copy of the AAA resource and sub-resources in another region that you must manually maintain.
  • I feel like there should be an option to automatically replicate these resources. This is so painful, especially if you are using credentials and certificates store in the AAA.

Runbooks with parameters

  • Runbooks can have parameters that you have to fill out before Runbook execution, great.
  • If you have a scheduled Runbook, the parameters are stored in the schedule resource. But if you want to kick off the schedule now, you have to manually copy and paste those parameter values.
    • This is fine for simple parameters, but if you have a Runbook that takes many parameters or for example accepts a JSON string. You need to copy and paste the values while removing any escaped characters (back-slashes in front of double quotes)
    • I feel like schedules should just have a "run now" button that fires off an execution with the same parameters.

Sandbox

# Ensures you do not inherit an AzContext in your runbook
Disable-AzContextAutosave -Scope Process
  • Also run all this code to make sure your Runbook isn't already running
# Ensures you do not inherit an AzContext in your runbook
Disable-AzContextAutosave -Scope Process

# Connect to Azure with system-assigned managed identity
$AzureContext = (Connect-AzAccount -Identity).context

# set and store context
$AzureContext = Set-AzContext -SubscriptionName $AzureContext.Subscription `
    -DefaultProfile $AzureContext

# Check for already running or new runbooks
$runbookName = "runbookName"
$resourceGroupName = "resourceGroupName"
$automationAccountName = "automationAccountName"

$jobs = Get-AzAutomationJob -ResourceGroupName $resourceGroupName `
    -AutomationAccountName $automationAccountName `
    -RunbookName $runbookName `
    -DefaultProfile $AzureContext

# Check to see if it is already running
$runningCount = ($jobs.Where( { $_.Status -eq 'Running' })).count

if (($jobs.Status -contains 'Running' -and $runningCount -gt 1 ) -or ($jobs.Status -eq 'New')) {
    # Exit code
    Write-Output "Runbook $runbookName is already running"
    exit 1
} else {
    # Insert Your code here
    Write-Output "Runbook $runbookName is not running"
}

AzureRM

  • The AzureRM PowerShell module should no longer be used, it is deprecated and will no longer exist at the end of Feb 2024.

Because Az PowerShell modules now have all the capabilities of AzureRM PowerShell modules and more, we'll retire AzureRM PowerShell modules on 29 February 2024.

To avoid service interruptions, update your scripts that use AzureRM PowerShell modules to use Az PowerShell modules by 29 February 2024. To automatically update your scripts, follow the quick-start guide.

  • Instead of AzureRM you should use the Az module.
    • You are not allowed to load the Az and AzureRM modules at the same time.
  • Microsoft has this to say about using AzureRM

Do not include the keyword "AzureRm" in any script designed to be executed with the Az module. Inclusion of the keyword, even in a comment, may cause the AzureRm to load and then conflict with the Az module.

  • To me that sounds like they do a plaintext search over your Runbook code, if they find "AzureRM" they load the module, breaking the Az module.

RBAC

  • This page has a table explaining the different roles and what they allow you to do https://learn.microsoft.com/en-us/azure/automation/automation-role-based-access-control
    • Do you want to limit which Runbooks users can run? It sounds like a pretty standard thing that should be super simple with Azure's great RBAC system.
    • Make sure you read this bit:

      You can only set role-based access control at the Automation account scope and not at any resource below the Automation account.

    • And then for a twist there is also this bit:

      If you want to control operator access to individual runbooks then don’t set this role. Instead use the Automation Job Operator and Automation Runbook Operator roles in combination.

    • Sounds good until you read this in the Automation Job Operator section:

      An Automation Job Operator role is granted at the Automation account scope. This allows the operator permissions to create and manage jobs for all runbooks in the account. If the Job Operator role is granted read permissions on the resource group containing the Automation account, members of the role have the ability to start runbooks. However, they don't have the ability to create, edit, or delete them.

    • This all means that if a user has access to run a single Runbook in that AAA, and they also have read access over the other Runbooks they are able to execute all Runbooks they have read access to in that AAA.
      • I think this is a bad system, personally I believe service desk and EUC teams should have read access to all Azure resources so they can better troubleshoot and diagnose issues before escalation. But in this case, it would mean they can execute Runbooks you may not want them to.
      • To get around this you could check who executed the Runbook inside the Runbook code and act accordingly, but then you re-inventing the RBAC wheel.

Source Control

  • The source control feature allows you to automatically sync Runbook code from a code repository. This is a great feature, never got to try it though because it fails completely if you have MFA/conditional access enabled.
    • I think it is better to push Runbook changes with a pipeline from DevOps/Github anyway.

Hybrid Runbook Worker (HRW) Specific Issues

AA has two execution methods for Runbooks.

  • Azure Sandbox
    • Azure sandbox is a SaaS (or PaaS?) service that will take your Runbook and execute it on an MS server in Azure somewhere.
  • Hybrid Runbook Worker
    • HRW allows you to set up your own execution environment, by installing an extension (or agent) on a VM.
    • Using a HRW means you can access resources on your own VNets and run anything else on that VM (e.g. run exe's from PowerShell)

Limited to a single AA

  • You should know that an HRW can only be connected to a single AAA. You cannot have a VM that serves as a HRW for multiple AAA.
  • If you want to have lots of AAA to split up Runbooks with RBAC, just remember you will need a VM per AAA.
  • This can be very expensive, especially if you want to have two HRWs per AAA for redundancy.

Onboarding with ARM

You are meant to be able to associate a VM to an AAA with an ARM template.

  • I tried to get this to work so I could deploy the whole solution as Infrastructure as Code (IaC) but was unable.
  • When researching the issue online, I found that many other people had this problem and the only way was through the portal.

Credentials for local resources

  • The HRW service runs with the local service account. This works fine for accessing resources on the HRW.
  • If you want to access network resources you need to store the credentials in the AAA first.
    • AAA uses a Key Vault in the backend but abstracts the interface away, giving you
  • I wanted to execute Runbooks under a Group Managed Service Account (gMSA) to access network resources
    • Changing the service log-on account is not supported
    • The recommended method from MS is:
      • Create a Runbook to use as a bootstrap
      • This Runbook should download the actual PowerShell script (from another Runbook resource)
      • The bootstrap runbook should execute psexec to execute the Runbook under the gMSA context

Run As Accounts with certificate authentication

Run As accounts are an older deprecated method of authenticating to resources. They work by storing account login details in the backend AAA Key Vault, the details can be retrieved with PowerShell.

  • I had a task to migrate runbooks from one AAA to a new AAA. The new AAA used a HRW.
  • If you want to use a Run As certificate on a HRW, you need to install the certificate into the certificate store on the HRW manually. AA will not do this for you.
  • Microsoft provides a PowerShell script you can run on AAAs as a Runbook to export certificates to a storage account.
  • I didn't want to have to set up a storage account for something that should be doable in the portal. So I found some code on StackOverflow that exported it to a base64 string.
$cert = Get-AutomationCertificate -Name "AzureRunAsCertificate"
$certData = $cert.Export("pfx", 'MySuperSecurePassword')
Write-Output $certData
  • Then you import like this
$b64 = @"
Put your Base64 string from the runbook on this line
"@
$pfx = [System.Convert]::FromBase64String($b64)
Set-Content -Value $pfx -Path "C:\temp\export.pfx" -Force -Encoding Byte

Just be mindful that this does use a plaintext password that may be available to anyone who has access to read the source code. But MS seems to think this is fine.

  • I feel like AA should install the certificates for you.
    • In future I will probably set up a pre-requisite script that automatically handles checking for and installation of certificates by calling a function at the start of a Runbook.
    • If you are using managed identities then this is a non-issue. Unless you are accessing some third-party applications that need certificates.

PowerShell Modules

Within your AAA resource, you can specify which modules (including versions) will be available to your Runbooks when executed.

  • There is no easy way to bulk update versions to the latest.
  • If you are using a HRW, you need to manually manage these modules yourself.
    • I feel like AA should manage this for you, if each AAA had it's own user context then the modules could be installed under that scope.
    • Instead modules need to be install to the AllUsers scope. Let's hope non of these modules/versions interfere with modules you use for management.

Watcher Tasks

Watcher Tasks are (were) another way to execute Runbooks.

  • They are meant to be used for things like firing off a Runbook when a file appears in a directory.
  • Essentially they are just Runbooks that run every 5 minutes or so.
  • I planned on using one of these, I thought they were great until I tried to actually use one and got an ambiguous error.
The remote server returned an error: (400) Bad Request.

Well it turns out that Watcher Tasks are no longer a thing. But the documentation doesn't say that. The only hint to that is this a notice on this documentation page https://learn.microsoft.com/en-us/azure/automation/automation-scenario-using-watcher-task

Starting in May 2020, using Azure Logic Apps is the recommended and supported way to monitor for events, schedule recurring tasks, and trigger actions. See Schedule and run recurring automated tasks, processes, and workflows with Azure Logic Apps

It doesn't say anything about them being deprecated or not working anymore. Searching online I found this Reddit thread of someone with the same problem. https://www.reddit.com/r/AZURE/comments/egey5t/azure_automation_watcher_task_error/

Follow-up: After discussing this with MS support, we were informed that the Watcher feature is being deprecated and they recommend a schedule-based trigger. Not ideal for all situations, but it is what it is. So, basically, move away from using a Watcher Task as soon as you can.