I run Terraform using a service principal which is simple enough to create.

I found the service principal had issues in creating AAD objects like application registrations or even querying the directory for objects when using data resources.

I was missing the correct Azure roles on the service principal and found a link that helped.

Create a file as below, editing what you need. The Id needs to be a unique GUID:

{
  "Name": "DevOps Contributor",
  "Id": "aaaaaaaa-bbbb-bbbb-bbbb-cccccccccccc",
  "IsCustom": true,
  "Description": "Grants full access to manage all resources and assign roles in Azure RBAC, but does not allow you to manage assignments in Azure Blueprints, or share image galleries.",
  "Actions": [
    "*"
  ],
  "NotActions": [
    "Microsoft.Blueprint/blueprintAssignments/write",
    "Microsoft.Blueprint/blueprintAssignments/delete",
    "Microsoft.Compute/galleries/share/action"
  ],
  "DataActions": [],
  "NotDataActions": [],
  "AssignableScopes": [
    "/subscriptions/aaaaaaaa-bbbb-bbbb-bbbb-cccccccccccc"
  ]
}

Now we can run a powershell command to apply this to Azure:

New-AzRoleDefinition -InputFile rbac.json

We can check it exists by running:

Get-AzRoleDefinition -Name "DevOps Contributor"

Now, we can go to the IAM tab of the Azure subscription to apply this custom Azure role to the service principal which will then be able to query and create Azure directory object.