Terraform

Introduction

References

• Azure providers and network
• Provider Configuration
• Terraform Course - Automate your AWS cloud infrastructure

This is an attempt to set-up a K8s cluster in Azure, using the information in compute resources

See terraform plugin at: Azure VM

Overview

1. terraform init
2. terraform plan
3. terraform apply

x. terraform destroy !!! it seems it also tries to delete the resource group :-(

• tf workspace -h
  • terraform workspace new NEW_WORKSPACE
  • terraform workspace list

File structure

• .terraform/ - gets create when any plugin is initialized
• terraform.tfstate - represents all the states for terraform
  • keeps track of how terraform believes the state is.
• .terraform.lock.hcl - keeps track of which module version was used.
  • to commit of not to commit
  • commit if you want the deployment unaffected by modul updates
  • do not commit it, if you want to know immediatly if a module update breakes the deployment.

Installation

See: [www.terraform.io/downloads.html](Install Terraform)

• wget -O- https://apt.releases.hashicorp.com/gpg | sudo gpg --dearmor -o /usr/share/keyrings/hashicorp-archive-keyring.gpg
• echo "deb [signed-by=/usr/share/keyrings/hashicorp-archive-keyring.gpg] https://apt.releases.hashicorp.com $(lsb_release -cs) main" | sudo tee /etc/apt/sources.list.d/hashicorp.list
• sudo apt update && sudo apt install terraform

Start a new project

• Providers list

Terraform has created a lock file .terraform.lock.hcl to record the provider
selections it made above. Include this file in your version control repository
so that Terraform can guarantee to make the same selections by default when
you run "terraform init" in the future.

Azure provider

• Overview of Terraform on Azure

• Azure Provider Hashicorp documentation

• Install Terraform on Windows with Bash

• Create the main.tf with the required provider, as per Azure Provider documentation

• terraform init

• git add .terraform.lock.hcl

• az login

• az account show

Libvirt provider

• Libvirt Provider

• Example

• terraform with libvirt

• using libvirt for terraform

libvirt data disk

resource "libvirt_volume" "data-qcow2" {
  name = "csgo-data.qcow2"
  pool = "default" # List storage pools using virsh pool-list
  format = "qcow2"
  size = 45097156608
}

The disk can be formated using the cloudinit, see 'prepare an extra disk'

libvirt example

# Configure the Libvirt provider

terraform {
  required_providers {
    libvirt = {
      source = "dmacvicar/libvirt"
      version = "0.7.1"
    }
  }
}

# https://registry.terraform.io/providers/dmacvicar/libvirt/latest

# See: https://registry.terraform.io/providers/multani/libvirt/latest/docs

provider "libvirt" {
  uri = "qemu:///system"
}

# Defining VM Volume
resource "libvirt_volume" "terraform-test-qcow2" {
  name = "terraform_test.qcow2"
  pool = "default" # List storage pools using virsh pool-list
  source = "/disk2/jammy-server-cloudimg-amd64.qcow2"
  format = "qcow2"
}

# Define KVM domain to create
resource "libvirt_domain" "terraform-test-vm" {
  name   = "tf_test_vm"
  memory = "2048"
  vcpu   = 2

  network_interface {
    network_name = "default" # List networks with virsh net-list
    bridge = "virbr0"
    wait_for_lease = true
  }

  disk {
    volume_id = "${libvirt_volume.terraform-test-qcow2.id}"
  }

  console {
    type = "pty"
    target_type = "serial"
    target_port = "0"
  }

  graphics {
    type = "spice"
    listen_type = "address"
    autoport = true
  }
}

# Output Server IP
output "ip" {
  value = "${libvirt_domain.terraform-test-vm.network_interface.0.addresses.0}"
}

See also:

• Using the Libvirt Provisioner With Terraform for KVM

Syntax

details

• resource
  • resource "TYPE" "INTERNAL_NAME"
  • TYPE: e.g. 'aws_instance'
  • INTERNAL_NAME: name used by this terraform file to reference this resource

Output

worker_node_ips = [
  {
    "arch" = "x86_64"
    "autostart" = false
    "boot_device" = tolist([])
    "cloudinit" = "/hdd2/vmstoragepool/k8s_lfs_worker_commoninit_0.iso;45574da4-726b-4d6b-ae9a-94b1fdae66f9"
    "cmdline" = tolist(null) /* of map of string */
    "console" = tolist([
      {
        "source_host" = "127.0.0.1"
        "source_path" = ""
        "source_service" = "0"
        "target_port" = "0"
        "target_type" = "serial"
        "type" = "pty"
      },
    ])
    "coreos_ignition" = tostring(null)
    "cpu" = tolist([
      {
        "mode" = "custom"
      },
    ])
    "description" = ""
    "disk" = tolist([
      {
        "block_device" = ""
        "file" = ""
        "scsi" = false
        "url" = ""
        "volume_id" = "/hdd2/vmstoragepool/k8s_lfs_worker_0.qcow2"
        "wwn" = ""
      },
    ])
    "emulator" = "/usr/bin/qemu-system-x86_64"
    "filesystem" = tolist([])
    "firmware" = tostring(null)
    "fw_cfg_name" = "opt/com.coreos/config"
    "graphics" = tolist([
      {
        "autoport" = true
        "listen_address" = "127.0.0.1"
        "listen_type" = "address"
        "type" = "spice"
        "websocket" = 0
      },
    ])
    "id" = "7a41abbd-e9e2-46cc-84d4-b3617677e394"
    "initrd" = ""
    "kernel" = ""
    "machine" = "ubuntu"
    "memory" = 4096
    "metadata" = tostring(null)
    "name" = "k8s_lfs_worker_0_vm"
    "network_interface" = tolist([
      {
        "addresses" = tolist([
          "192.168.122.226",
        ])
        "bridge" = ""
        "hostname" = ""
        "mac" = "52:54:00:E6:E1:A9"
        "macvtap" = ""
        "network_id" = "e26992cd-dc45-4f9c-bb9b-75ba01de9ca7"
        "network_name" = "default"
        "passthrough" = ""
        "vepa" = ""
        "wait_for_lease" = true
      },
    ])
    "nvram" = tolist([])
    "qemu_agent" = false
    "running" = true
    "timeouts" = null /* object */
    "tpm" = tolist([])
    "type" = "kvm"
    "vcpu" = 2
    "video" = tolist([])
    "xml" = tolist([])
  },
  {
    "arch" = "x86_64"
    "autostart" = false
    "boot_device" = tolist([])
    "cloudinit" = "/hdd2/vmstoragepool/k8s_lfs_worker_commoninit_1.iso;36f42771-4372-413c-8eb3-f6210106074d"
    "cmdline" = tolist(null) /* of map of string */
    "console" = tolist([
      {
        "source_host" = "127.0.0.1"
        "source_path" = ""
        "source_service" = "0"
        "target_port" = "0"
        "target_type" = "serial"
        "type" = "pty"
      },
    ])
    "coreos_ignition" = tostring(null)
    "cpu" = tolist([
      {
        "mode" = "custom"
      },
    ])
    "description" = ""
    "disk" = tolist([
      {
        "block_device" = ""
        "file" = ""
        "scsi" = false
        "url" = ""
        "volume_id" = "/hdd2/vmstoragepool/k8s_lfs_worker_1.qcow2"
        "wwn" = ""
      },
    ])
    "emulator" = "/usr/bin/qemu-system-x86_64"
    "filesystem" = tolist([])
    "firmware" = tostring(null)
    "fw_cfg_name" = "opt/com.coreos/config"
    "graphics" = tolist([
      {
        "autoport" = true
        "listen_address" = "127.0.0.1"
        "listen_type" = "address"
        "type" = "spice"
        "websocket" = 0
      },
    ])
    "id" = "edea633c-35e4-4115-8c33-edbd8c90537e"
    "initrd" = ""
    "kernel" = ""
    "machine" = "ubuntu"
    "memory" = 4096
    "metadata" = tostring(null)
    "name" = "k8s_lfs_worker_1_vm"
    "network_interface" = tolist([
      {
        "addresses" = tolist([
          "192.168.122.91",
        ])
        "bridge" = ""
        "hostname" = ""
        "mac" = "52:54:00:76:0A:5A"
        "macvtap" = ""
        "network_id" = "e26992cd-dc45-4f9c-bb9b-75ba01de9ca7"
        "network_name" = "default"
        "passthrough" = ""
        "vepa" = ""
        "wait_for_lease" = true
      },
    ])
    "nvram" = tolist([])
    "qemu_agent" = false
    "running" = true
    "timeouts" = null /* object */
    "tpm" = tolist([])
    "type" = "kvm"
    "vcpu" = 2
    "video" = tolist([])
    "xml" = tolist([])
  },
]

Providers

LibVirt

Azure

# https://registry.terraform.io/providers/hashicorp/azurerm/latest/docs


# Create the service principal to use for the terraform operations
#   https://learn.microsoft.com/en-us/cli/azure/create-an-azure-service-principal-azure-cli
# First create a csgo resource group, via the web interface
# For the rolle look at the 'Access control(IAM)' entry under the resource group web page.
#  Contributor -  Grants full access to manage all resources, but does not allow you to assign roles in Azure RBAC, manage assignments in Azure Blueprints, or share image galleries.
# subscriptionID=$(az account show --query id -o tsv)
# mkdir ~/.variouos_credentials && chmod 700 ~/.variouos_credentials
# az ad sp create-for-rbac --name terraform_op --role Contributor --scopes /subscriptions/$subscriptionID/resourceGroups/$resourceGroup > ~/.variouos_credentials/azure.$resourceGroup.terraform_op

# add the following to your private apply script, do not check this in
# export ARM_SUBSCRIPTION_ID="<azure_subscription_id>"
# export ARM_TENANT_ID="<azure_subscription_tenant_id>"
# export ARM_CLIENT_ID="<service_principal_appid>"
# export ARM_CLIENT_SECRET="<service_principal_password>"

Azure storage

Azure storage acccount

Azure file shares

• Create an SMB Azure file share

ESXi provider

Variables

Defining variables

variable "csgo_client_access_password" {
  description = "Password clients use to connect to the server"
  default = "VerySecret"
}

Using variables

var.csgo_client_access_password

Providing variables at the command line call

terraform apply -var csgo_client_access_password="WelcomeToMyServer"

file structure

• main.tf - the configuration for the resources that you want to create.
• network.tf - common network definitions.
• outputs.tf - what you want output on after the apply.
• providers.tf - definition of the provider. What terraform provider to use.
• private_azure_variables.tf - private information, not to be included in the source control.
  • e.g. tokens etc.
• README.md
• server-init-actions.yaml - cloud init file, one per server type.
• server.tf - server definition, one per server type.
• variables.tf - values that are likely to be changed.

Cloudinit

• libvirt example
• cloudinit Provider
• Cloud config examples

Cloud-init commands are run when a VM boots for the first time.

Them commands and files are packed onto an ISO file and that ISO file is mounted by the VM at first boot. Then a std command is called on the ISO and the operations are then executed.

Using cloud init in terraform

You need the following

Define the YAML with the cloud-init actions

#cloud-config
# Add groups to the system
groups:
  - hashicorp

# Add users to the system. Users are added after groups are added.
users:
  - default
  - name: ansible
    gecos: ansible
    shell: /bin/bash
    primary_group: hashicorp
    sudo: ALL=(ALL) NOPASSWD:ALL
    groups: users, admin
    lock_passwd: false
    ssh_authorized_keys:
      - ${ansible_ssh_public_key}

# install packages
packages:
  - ansible
  - git

# create format a second disk
disk_setup:
  /dev/vdb:
    table_type: 'gpt'
    layout: True
    overwrite: false
fs_setup:
  - label: data
    device: /dev/vdb1
    filesystem: ext4
    overwrite: false
mounts:
  - [/dev/vdb1, /data, ext4, 'defaults,discard,nofail', '0', '2']

# commands to run at the end
runcmd:
  - cd /home/ansible; git clone  https://github.com/XXX/ansible_instructions
  - cd ansible_instructions && ansible-playbook -v some_playbook.yaml

Define the cloud-init template

data "template_file" "user_data" {
  template = file("./cloud-init-actions.yaml")

  # left hand var names are the var names used in the cloud-init yaml.
  vars = {
    ansible_ssh_public_key = file(var.ansible_ssh_public_key_filename)
    csgo_client_access_password = var.csgo_client_access_password
    csgo_server_rcon_password = var.csgo_server_rcon_password
    one_for_local_zero_for_global = var.one_for_local_zero_for_global
    server_name = var.server_name
    steam_server_token = var.steam_server_token
  }
}

Generate the cloud-init ISO file

resource "libvirt_cloudinit_disk" "commoninit" {
  name      = "csgo_commoninit.iso"
  user_data = data.template_file.user_data.rendered
}

Reference the cloud-init ISO file in the domain(vm) definition

resource "libvirt_domain" "csgo-vm" {
  ...
  cloudinit = libvirt_cloudinit_disk.commoninit.id
  ...
}

Tranferring variables from terraform to cloud-init

• Dynamic Cloud-Init Content with Terraform File Templates
  • terraform-vsphere-cloudinit-dynamic
• Terraform : passing variables from resource to cloudinit data block

data "template_file" "user_data" {
  template = file("./cloud-init-actions.yaml")

  # left hand var names are the var names used in the cloud-init yaml.
  vars = {
    ansible_ssh_public_key = file(var.ansible_ssh_public_key_filename)
    csgo_client_access_password = var.csgo_client_access_password
    csgo_server_rcon_password = var.csgo_server_rcon_password
    one_for_local_zero_for_global = var.one_for_local_zero_for_global
    server_name = var.server_name
    steam_server_token = var.steam_server_token
  }
}

Passing cloud init variables onto script

ansible-playbook --extra-vars "ansible_var_namecsgo_client_access_password=${cloud_init_var_name_csgo_client_access_password}"

transferring files using cloud-init files

copy a local file to the VM being deployed.

• cloud-init part

# provide the CNI deployment file.
#  https://cloudinit.readthedocs.io/en/latest/reference/modules.html#write-files
write_files:
  - path: /home/ansible/k8s_node_ansible_playbook.yaml
    content: !!binary |
      ${ansible_playbook_file_base64}
    permissions: '0640'

• ansible_playbook_file_base64 is defined in the .tf file.

  • worker.tf: ansible_playbook_file_base64 = filebase64("../../ansible_playbook/k8s_node_ansible_playbook.yaml")
  • in the template_file section
    • data "template_file" "worker_user_data" {

• terraform part

data "template_file" "control_plane_user_data" {
  template = file("./cloud-init-actions.yaml")

  # left hand var names are the var names used in the cloud-init yaml.
  vars = {
    ansible_ssh_public_key = file(var.ansible_ssh_public_key_filename)
    node_type = "control_plane"
    cni_installation_file_base64 = filebase64("weave-daemonset-k8s.yaml")
  }
}

Cloudinit disk operations

prepare an extra disk

disk_setup:
  /dev/vdb:
    table_type: 'gpt'
    layout: True
    overwrite: false
fs_setup:
  - label: data
    device: /dev/vdb1
    filesystem: ext4
    overwrite: false
mounts:
  - [/dev/vdb1, /data, ext4, 'defaults,discard,nofail', '0', '2']

Troubleshooting

troubleshooting libvirt

Error: failed to dial libvirt: ssh: handshake failed: knownhosts: key mismatch

See. Connection to libvirt via SSH fails with provider version 0.6.9

• Can you ssh to the server without challenge?
  • Probably not, you need to be able to e.g. ssh [email protected]
• Deleteing id_rsa and ~/.ssh/known_hosts fixed the issue
  • not sure if deleting known_hosts would have been enough.

│ Error: failed to dial libvirt: ssh: handshake failed: knownhosts: key mismatch
│ 
│   with provider["registry.terraform.io/dmacvicar/libvirt"],
│   on main.tf line 16, in provider "libvirt":
│   16: provider "libvirt" {
│ 

domain 'cloudstack_vm' already exists with uuid b63b2a6f-2dc0-493f-af8d-5a5a7561a97a

virsh list --all virsh undefine cloudstack_vm

libvirt_volume.os-qcow2: Creation complete after 1m13s [id=/var/lib/libvirt/images/tf_cloudstack.qcow2]
libvirt_cloudinit_disk.commoninit: Creation complete after 1m13s [id=/var/lib/libvirt/images/cloudstack_commoninit.iso;e5958a5a-8adc-4584-92f7-f7a8a617bb68]
libvirt_domain.cloudstack-vm: Creating...
╷
│ Error: error defining libvirt domain: operation failed: domain 'cloudstack_vm' already exists with uuid b63b2a6f-2dc0-493f-af8d-5a5a7561a97a
│ 
│   with libvirt_domain.cloudstack-vm,
│   on main.tf line 52, in resource "libvirt_domain" "cloudstack-vm":
│   52: resource "libvirt_domain" "cloudstack-vm" {
│ 

Could not open '/var/lib/libvirt/images/terraform_test.qcow2': Permission denied

Solution: How to fix Terraform libvirt permission denied on /var/lib/libvirt/images

• sudo vi /etc/apparmor.d/libvirt/TEMPLATE.qemu
• sudo systemctl restart libvirtd

│   with libvirt_domain.terraform-test-vm,
│   on template.tf line 31, in resource "libvirt_domain" "terraform-test-vm":
│   31: resource "libvirt_domain" "terraform-test-vm" {

#
# This profile is for the domain whose UUID matches this file.
#

#include <tunables/global>

profile LIBVIRT_TEMPLATE flags=(attach_disconnected) {
  #include <abstractions/libvirt-qemu>
  /hdd2/vmstoragepool/**.qcow2 rwk,
}

original

#
# This profile is for the domain whose UUID matches this file.
#

#include <tunables/global>

profile LIBVIRT_TEMPLATE flags=(attach_disconnected) {
  #include <abstractions/libvirt-qemu>
}

Error: error creating libvirt domain: internal error: process exited while connecting to monitor: 2023-09-07T08:39:13.291493Z qemu-system-x86_64: -blockdev {"driver":"file","filename":"/var/lib/libvirt/images/tf_cloudstack.qcow2","node-name":"libvirt-2-storage","auto-read-only":true,"discard":"unmap"}: Could not open '/var/lib/libvirt/images/tf_cloudstack.qcow2': Permission denied

libvirt_domain.gitlab-os-vm: Creating...
╷
│ Error: error creating libvirt domain: internal error: process exited while connecting to monitor: 2023-11-07T08:34:22.172477Z qemu-system-x86_64: -blockdev {"driver":"file","filename":"/hdd2/vmstoragepool/gitlab.qcow2","node-name":"libvirt-1-storage","auto-read-only":true,"discard":"unmap"}: Could not open '/hdd2/vmstoragepool/gitlab.qcow2': Permission denied
│ 
│   with libvirt_domain.gitlab-os-vm,
│   on gitlab.tf line 14, in resource "libvirt_domain" "gitlab-os-vm":
│   14: resource "libvirt_domain" "gitlab-os-vm" {
│ 
╵```

#### Error: error retrieving volume for disk: Storage volume not found: no storage vol with matching name 'csgo.qcow2'

See: [Error retrieving volume for disk](https://github.com/dmacvicar/terraform-provider-libvirt/issues/949)

* rm terraform.tfstate terraform.tfstate.backup

#### Error: error creating libvirt domain: internal error: process exited while connecting to monitor Could not open qcow2 Permission denied

[](https://github.com/dmacvicar/terraform-provider-libvirt/issues/715)

[](https://computingforgeeks.com/use-virt-manager-as-non-root-user/)

sudo usermod -a -G libvirt-qemu cadm

```text
│ Error: error creating libvirt domain: internal error: process exited while connecting to monitor: 2023-09-11T17:22:29.489038Z qemu-system-x86_64: -blockdev {"driver":"file","filename":"/var/lib/libvirt/images/lfs_worker_0.qcow2","node-name":"libvirt-2-storage","auto-read-only":true,"discard":"unmap"}: Could not open '/var/lib/libvirt/images/lfs_worker_0.qcow2': Permission denied

libvirt_volume.os-qcow2: Creation complete after 56s [id=/hdd2/vmstoragepool/tf_cloudstack.qcow2]
libvirt_domain.cloudstack-vm: Creating...
╷
│ Error: error creating libvirt domain: internal error: process exited while connecting to monitor: 2023-09-12T14:24:30.727925Z qemu-system-x86_64: -blockdev {"driver":"file","filename":"/hdd2/vmstoragepool/tf_cloudstack.qcow2","node-name":"libvirt-1-storage","auto-read-only":true,"discard":"unmap"}: Could not open '/hdd2/vmstoragepool/tf_cloudstack.qcow2': Permission denied
│ 
│   with libvirt_domain.cloudstack-vm,
│   on main.tf line 33, in resource "libvirt_domain" "cloudstack-vm":
│   33: resource "libvirt_domain" "cloudstack-vm" {
│ 
╵

troubleshoot azure

Error: building account: could not acquire access token to parse claims: clientCredentialsToken: failed to build request

Planning failed. Terraform encountered an error while generating this plan.

╷
│ Error: building account: could not acquire access token to parse claims: clientCredentialsToken: failed to build request: parse "https://login.microsoftonline.com/XXXXXXXXXXX\n/oauth2/v2.0/token": net/url: invalid control character in URL
│ 
│   with provider["registry.terraform.io/hashicorp/azurerm"],
│   on main.tf line 32, in provider "azurerm":
│   32: provider "azurerm" {

Error: Error ensuring Resource Providers are registered

TODO what is "register Azure Resource Providers." see: Azure Provider Note

??? Is it this thing of storring the state in azure cloud?

Add

provider "azurerm" {
  skip_provider_registration = true # This is only required when the User, Service Principal, or Identity running Terraform lacks the permissions to register Azure Resource Providers.
}

│ Error: Error ensuring Resource Providers are registered.
│ 
│ Terraform automatically attempts to register the Resource Providers it supports to
│ ensure it's able to provision resources.
│ 
│ If you don't have permission to register Resource Providers you may wish to use the
│ "skip_provider_registration" flag in the Provider block to disable this functionality.
...
│ More information on the "skip_provider_registration" flag can be found here:
│ https://registry.terraform.io/providers/hashicorp/azurerm/latest/docs#skip_provider_registration

The requested size for resource is currently not available in location

Turns out the Basic_A0 was not available in that location

az vm list-skus --location northeurope --size Basic_ --all --output table

azurerm_network_interface_security_group_association.csgo_nsg_association: Creation complete after 1s [id=/subscriptions/XXX/resourceGroups/free_csgo_group/providers/Microsoft.Network/networkInterfaces/csgoNIC|/subscriptions/XXX/resourceGroups/free_csgo_group/providers/Microsoft.Network/networkSecurityGroups/csgo-nsg]
╷
│ Error: creating Linux Virtual Machine: (Name "csgoVM" / Resource Group "free_csgo_group"): compute.VirtualMachinesClient#CreateOrUpdate: Failure sending request: StatusCode=409 -- Original Error: Code="SkuNotAvailable" Message="The requested size for resource '/subscriptions/XXX/resourceGroups/free_csgo_group/providers/Microsoft.Compute/virtualMachines/csgoVM' is currently not available in location 'northeurope' zones '' for subscription 'XXX'. Please try another size or deploy to a different location or zones. See https://aka.ms/azureskunotavailable for details."
│ 
│   with azurerm_linux_virtual_machine.csgo_vm,
│   on main.tf line 107, in resource "azurerm_linux_virtual_machine" "csgo_vm":
│  107: resource "azurerm_linux_virtual_machine" "csgo_vm" {

Troubleshooting the azure provisioner

Inappropriate value for attribute "destination_port_range": string required

this works: destination_port_range = "27015-27017"

│ Error: Incorrect attribute value type
│ 
│   on main.tf line 119, in resource "azurerm_network_security_group" "csgo_nsg":
│  119:     destination_port_range     = ["27015-27017"]
│ 
│ Inappropriate value for attribute "destination_port_range": string required.